Batch V5 *NEW*
Version 5 of the Black Diamond Batch API is our newest batch offering featuring new functionality and a multitude of enhancements to assist with a variety of workflow needs. All previous batch version parameters will be supported, but a change to the request is required in order to retrieve the same data. *URLencoded is no longer supported on this version. JSON is now the required format.
Batch Output Files
The batch API consolidates hundreds of various data points across the Black Diamond platform into a single, compressed zip file to make pulling large data sets more efficient. This workflow is most commonly used to pull data sets for an entire firm or user on a daily or other regular frequency. The output from the Batch API is a zip file containing JSON-formatted data files grouped by the following entities:
Relationship - Relationships define ownership at a household level and are the top-level entity for all maintenance in Black Diamond
RelationshipV2 - This is a more consolidated and recommended version of the relationship file that is more in line with how relationships are represented within Black Diamond today. This trims out a lot of excess data and is more focused on entity association.
Portfolio - Consists of a group of accounts. A single account can exist in many portfolios and a portfolio may be associated with a Relationship.
Portfolio Group - A group of portfolios.
Account - A financial account record and its associated information
The account file is split into separate JSON files grouped by 250 accounts at a time.
Additional account level data points such as current holdings, transactions, and cost basis information will need to be explicitly asked for during the request.
Target - A defined allocation model that is used to compare against the current holdings in an account or portfolio. Not typically used for trading purposes.
Client - Represents users on the client portal who are investors and have a login to the Black Diamond Client Portal.
Contact - Contacts are used to track current and potential clients that may not need a login to Black Diamond.
Firm User - Represent advisors and administrators who have a login to Black Diamond.
Asset - All securities belonging to the firm as shown on the Security Master page as well as any asset level custom tags defined by the firm.
Workflow Overview:
Authenticate - Authenticate the user to the API using the workflows defined in our Authentication section here to get a valid Access Token
Initiate Batch Request - Use your Access Token to make a Post to the Batch API with the required parameters as defined below
Receive Batch ID - That API call kicks off the generation of the batch file on Black Diamond’s platform and returns a batch ID in the response to your call.
Monitor & Retrieve Batch Data - Monitor the status of the batch and use that batch ID to retrieve the batch file when ready.
Authenticate User
Authenticate your user using the Auth Code flow defined here or a previously acquired refresh token.
Initiate Batch Request
Using the acquired access token, make a https POST call to the batch end point: https://api.blackdiamondwealthplatform.com/batchV5
Reminder, all request headers must always include:
Authorization: Bearer {{AccessToken}}
OCP-APIM-Subscription-Key: {{SubscriptionKeyFromPortal}}
Content-Type: application/json
Required Parameters:
ReturnDate - The As of Date that the data file will be generated for
batchFiles - provides the ability to limit which files are received in the batch to reduce the batch size if all files are not needed. **By default, no files are generated unless specifically requested.**
Accepted values are: "Accounts", "Portfolio", "Relationship", "RelationshipV2", "Client", "Target", "Goals", "Contact", "FirmUser", "Asset"
In general, you only "Relationship" or "RelationshipV2" should be requested and not both.
Note: if specifying "Portfolio", this will also bring back the associated "PortfolioGroup" json file.
Parameter Format:
"batchFiles":["Accounts", "RelationshipV2"]
Optional Parameters:
By default, optional parameters are not included and will need to be opted into/requested in order for the information to be generated.
IncludeClosedAccounts - A true/false value that determines if accounts currently marked as closed on Black Diamond are included in the returned results. If not specified, defaults to false.
IncludeRecentlyClosedAccounts - includes closed accounts within the last 7 days based on the “returnDate” passed. *ignored if “
IncludeClodedAccounts” = “true". If not specified, defaults to false.
IncludeCostBasis - A true/false value to determine if you want to include account level cost basis and tax lot information.
IncludeRecentlyClosedLots - show tax lots closed within 30 days of requested ”
returnDate.” If not specified, defaults to NO closed lots in batch. *ignored if “includeCostBasis” = “false”CostBasisByHolding - A true/false to break down cost basis at a holding level.
IncludeTransactions - A true/false value that determines if account level transactions are returned. Default returns last 7 days of transactions based up on the specified “
returnDate”. Only one of the following sub transaction parameters can be requested at a time:SingleDayTransactions - returns only the transactions on the specified “
returnDate”. *ignored if “includeTransactions” = “false”TransactionDayLimit - determines how many calendar days to look back to pull in transactions based upon the specified “
returnDate”. Accepted values are [1-31]. *ignored if “includeTransactions” = “false”MonthtoDateTransactions - A true/false value to return month to date transactions based upon the requested “returnDate”. *ignored if “
includeTransactions” = “false”
IncludeHoldings - A true/false value that determines if account level holdings are returned. If not specified, defaults to false.
IncludeAllocation - A true/false value that determines if account class and segment allocation are returned. If not specified, defaults to false.
TargetPointsOfPrecision - 1-6, defaults to 4 and directly impacts ToleranceLower and ToleranceUpper
resultCallbackUrl - the batch API provides the ability to be notified via a webhook when the Batch file is complete and ready to be downloaded. The value provided in this parameter should be the full URL for the API to call back once your batch has completed. See below section on Accessing Batch Data for more information
clientToken - *Recommended if using a Callback URL* the client token is a value that should be generated per call and should be unique. BlackDiamond will store this token and send it to the provided callback URL; this helps ensure the call to your URL is a call you expected and not some arbitrary call. We recommend you compare this value to ensure the call to your callback end point is valid.
Optional Batch By Data Format Parameters:
These parameters are often used in conjunction with the firm status endpoint. More information can be found later in the article under the "How do I call the firm status endpoint?" section.
DataProviderIds - A true/false value to filter the account.json files to just the specified data formats/providers. If not specified, defaults to all data formats.
Format- DataProviderIds: [123, 456, 12340]
Please note that you should use the DataProviderIds that are provided from the Firm Recon Status V2 endpoint. These are Data Recon Ids and can encompass multiple account data providers Ids. For example, the DataProviderId returned from the Firm Recon Status endpoint for Fidelity IWS could be 124 and/or 1247, but both of these would translate to account data provider Id 31 within the account.json file.
IsReconciled - A true/false value to only include accounts that have completed reconciliation. If used in conjunction with the DataProviderIds parameter then only the specified data formats that have completed reconciliation will generate account information. If not provided, default is false. Additionally, closed accounts would be excluded.
DataProviderDelta - A true/false value used to return delta account files based on newly reconciled data providers. Please see Delta Data Format Best Practices below for more info. If not provided, default is false.
Delta Data Provider Best Pratices
The "DataProviderDelta" parameter is designed to be used in conjunction with the "IsReconciled" parameter and optionally can be used in conjunction with the "DataProviderIds" parameter. This parameter compares the user's previously run batches for the same specified "returnDate" in order to only return newly reconciled accounts/data providers. *Note that exception accounts can still be present in reconciled data providers.
Conditions:
DataProviderDelta:true- For the first batch of a given day, there will not be a prior batch to compare against and this parameter will be ignored.DataProviderDelta:trueandIsReconciled:true- For this example, let’s say the Black Diamond firm has just two data providers(Schwab and Pershing). Batch 1 will return all data providers and their respective accounts that have completed reconciliation at the time of the request. For this example, Schwab has completed recon but Pershing has not yet finished; therefore, Batch 1 will only contain Schwab accounts. Batch 2 is then run an hour later and Pershing has since completed recon. Batch 2 will only contain Pershing accounts(net new reconciled data providers). This same logic would continue to apply to subsequent batches, so Batch 3 would compare data providers against Batch 1 and Batch 2 and filter those respective data providers out of the response.DataProviderDelta:trueandIsReconciled:true- If you make a follow up request with these parameters and no new data formats have completed recon then we will return a 204 No Content status code indicating that no new data formats are available since the last request(s).DataProviderDelta:false- This is the default behavior if it is not included in the request. Accounts will not be filtered out based off previous batches. *Note the batch will still be filtered based upon the other parameters that are requested.
Optional Batch Encryption:
By default, batch files are not encrypted but users have the ability to provide a public key in the batch request body in order to PGP encrypt the zip file in its entirety. After retrieving the encrypted zip file, the user will need to decrypt using their private key.
Request Body Format:
"encryptionKey":{"publicKey": "-----BEGIN PGP PUBLIC KEY BLOCK-----PublicKeyValueHere-----END PGP PUBLIC KEY BLOCK,
"keyType": "PGP" }
The only accepted value for the “keyType” parameter is “PGP”.
The “publicKey” must be active and valid. Only the following encryption algorithms are supported.
RSA Sign Only
DSA
Elgamal
ECC Sign and Encrypt
Non-supported algorithms or invalid keys will result in a 403: “Provided PGP public key is not valid” error.
Requests can be made with or without the “BEGIN PGP PUBLIC KEY BLOCK” prefix and “END PGP PUBLIC KEY BLOCK” suffix. The "publicKey" needs to be json formatted properly including line breaks where applicable.
Example Request Body
{
"returnDate": "01-01-2024",
"batchFiles": ["Accounts"],
"includeCostBasis": true,
"includeTransactions": true,
"TransactionDayLimit": 31,
"includeHoldings": true,
"includeAllocation": true,
}
This above example would return similar information to our previous batch versions, but please modify with additional batch files or parameters to fit your use case. Highlighted fields are required and a minimum of one batch file type will need to be provided.
Receive Batch ID
If a successful post is made, the API with respond with the id for your specific batch request. Sample POST response from /batch:
{
"id": "826217bd-b0c7-40f6-a1db-5c3e4229399b"
}
Monitor & Retrieve Batch Data
Once a batch POST has been initiated, the time to generate the file will vary by firm depending on number of accounts. The typical file generation time is a few minutes but can run up to an hour for very large firms. There are two ways to retrieve the batch data when ready.
Utilize a Webhook Notification
Poll the Batch Endpoint
When a successful call is made to the GET /batch/{BatchId} the batch zip file will be streamed to you.
Webhook Notification
Black Diamond supports the concept of a “webhook” or “call back” URL. This means that once the batch file completes and is ready for download, Black Diamond will notify you of completion via the provided URL. A subsequent call is required to download the file, but the webhook eliminates the need to poll the GET /batch/{BatchId} endpoint until completion.
If a callback URL is provided as a parameter in the Batch post, Black Diamond will make a post to that URL when the batch file is finished generating with the below structure:
{
"batchRef": "826217bd-b0c7-40f6-a1db-5c3e4229399b",
"fileID": 70258143,
"returnDate": "2021-12-31",
"includeClosedAccounts": true,
"status": "Finished",
"callBackUri": "http://www.yourcallbackurlhere/",
"clientToken": "SomeUniqueValue",
"responseType": "Json"
}
When that notification is received you would make a GET call to /batch/{BatchId} to receive the zip file.
Poll Batch Endpoint
If you do not want to utilize the webhook notification, you can poll the GET call at /batch/{BatchId} to monitor the status of the batch file. The typical file generate time is a few minutes but can run up to an hour for very large firms. The response codes you may receive when polling this end point are detailed below.
FAQ
When will the data be ready each morning to be pulled?
Black Diamond is a T-1 system and thus receives account data files overnight from our data sources and reconciles that information each morning. We are dependent on the timing of receiving that data each morning and so when the data is ready for each firm will vary and is dependent on which data sources or custodians they utilize.
However we have a commitment to have all accounts through automated reconciliation by 9:00AM EST and on average most firms’ data becomes available between 7 and 8 am EST. On a daily basis, about 99% of accounts clear through automated reconciliation and about 1% of accounts are held out as exceptions for manual review. We do not have a specific time to clear all exceptions in a day. (generally it is clear between 9-11am ET and is always done by the end of the day)
We strongly recommend you utilize our Firm Status get call to monitor when a user or firms data has been updated through the previous business day.
***As of September 2023, the Black Diamond reconciliation process was enhanced to allow for accounts to be updated(T-1) on a data format level rather than a firm level. Previously, account as of dates would not be updated until recon for all custodians at the firm had finalized, but this enhancement will now update the account as of dates as soon as that particular custodian's recon is completed allowing you to potentially get data even earlier. The Firm Status endpoint has historically been used to check the recon status at a firm level, but it can be utilized to look at an individual custodian's recon completion time as well.
How do I call the firm status endpoint?
The firm status call can be accessed via the API section of this site under Firm API --> POST Firm Recon Status V2. Here are the different status responses you can expect:
Not Started – Black Diamond’s automated reconciliation process has not begun.
In Progress – Some but not all data providers have been processed through the automated reconciliation engine. Once all data providers have been competed the firm status will update from In Progress to Complete.
Complete (Exceptions) – **This is the status when the data becomes available for users to see on the Black Diamond UI** All available data providers have been completed through Black Diamond’s automated reconciliation engine. Exception is displayed due to an account not passing Black Diamond’s automated audits and now needs attention from a member of the operations team.
Complete (Aggregator) – All non-aggregated data providers have been completed through the Black Diamond automated reconciliation engine and no exceptions exists. The following providers are considered aggregated data providers and typically arrive later in the day (Plaid, ByAll Accounts).
Completed - All available data providers have been completed through Black Diamond’s automated reconciliation engine and there are no exceptions.
Sample response to the Firm Status V2 call:
{
"asOfDate": "2024-02-29T00:00:00Z",
"finalizeCompleteTime": "2024-03-01T12:30:43.933Z",
"firmReconStatus": "Complete (Aggregator)",
"totalExceptions": 0,
"dataProviderDetails": [{
"dataProviderName": "ByAllAccounts",
"dataProviderId": 156,
"exceptions": [],
"lastCompletedRGLDate": null,
"lastCompletedUGLDate": null,
"reconciliationCompleteTime": null,
"requiredForFinalize": false,
"status": "Not Started",
"totalExceptions": 0
}, {
"dataProviderName": "Vanguard - Flagship ADF",
"dataProviderId": 1221,
"exceptions": [],
"lastCompletedRGLDate": null,
"lastCompletedUGLDate": null,
"reconciliationCompleteTime": "2024-03-01T02:04:49Z",
"requiredForFinalize": true,
"status": "Complete",
"totalExceptions": 0
}, {
"dataProviderName": "Schwab",
"dataProviderId": 551,
"exceptions": [],
"lastCompletedRGLDate": "2024-02-29T00:00:00Z",
"lastCompletedUGLDate": "2024-02-29T00:00:00Z",
"reconciliationCompleteTime": "2024-03-01T09:03:22Z",
"requiredForFinalize": true,
"status": "Complete",
"totalExceptions": 0
}, {
"dataProviderName": "Fidelity IWS",
"dataProviderId": 1247,
"exceptions": [],
"lastCompletedRGLDate": null,
"lastCompletedUGLDate": null,
"reconciliationCompleteTime": "2024-03-01T09:03:34Z",
"requiredForFinalize": true,
"status": "Complete",
"totalExceptions": 0
}, {
"dataProviderName": "Fidelity CGF",
"dataProviderId": 196,
"exceptions": [],
"lastCompletedRGLDate": null,
"lastCompletedUGLDate": null,
"reconciliationCompleteTime": "2024-03-01T09:46:12Z",
"requiredForFinalize": true,
"status": "Complete",
"totalExceptions": 0
}
]}
What are the response codes when trying to download a batch file?
Response Codes:
200: File is ready and downloading.
202: You have submitted a batch that is currently in progress. Please check back later.
204: Returned when no new files are available when using delta file requests.
401: Token is expired or the user does not have access to retrieve the data you are seeking anymore
403: Missing terms and conditions. This user has to log in and accept the T&C.
404: Standard not found implementation. Following points of clarification:
Batch is purged or currently unavailable.
404 is also returned when the path or values are incorrect, missing, or not found.
429: Rate limited. Too many requests in a given time period. The rate limits will be set appropriately for you given that you’ll be submitting a batch per user. If running into this consistently please reach out to your Black Diamond contact.
500: This is an unexpected error. It should return a correlation ID along with the error. Providing this correlation ID to Black Diamond helps us troubleshoot the concern.