Batch V5 *Coming Soon*
Version 5 of the Black Diamond Batch API is our newest batch offering featuring various enhancements and new functionality to assist with a variety of workflow needs. All previous batch version options will be supported, but will require a change in the request in order to retrieve the same data.
If you are interested in being an early adopter for the BETA product then please reach out to the appropriate resource: https://www.blackdiamondwealthplatform.com/contactus
Batch Output Files
The batch API consolidates hundreds of various data points across the Black Diamond platform into a single, compressed zip file to make pulling large data sets more efficient. This workflow is most commonly used to pull data sets for an entire firm or user on a daily or other regular frequency. The output from the Batch API is a zip file containing JSON-formatted data files grouped by the following entities:
Relationship - Relationships define ownership at a household level and are the top-level entity for all maintenance in Black Diamond
Portfolio - Consists of a group of accounts. A single account can exist in many portfolios and a portfolio may be associated with a Relationship.
Portfolio Group - A group of portfolios.
Account - A financial account record and its associated information
The account file is split into separate JSON files grouped by 250 accounts at a time.
Additional account level data points such as current holdings, transactions, and cost basis information will need to be explicitly asked for during the request.
Target - A defined allocation model that is used to compare against the current holdings in an account or portfolio. Not typically used for trading purposes.
Client - Represents users on the client portal who are investors and have a login to the Black Diamond Client Portal.
Contact - Contacts are used to track current and potential clients that may not need a login to Black Diamond.
Firm User - Represent advisors and administrators who have a login to Black Diamond.
Workflow Overview:
Authenticate - Authenticate the user to the API using the workflows defined in our Authentication section here to get a valid Access Token
Initiate Batch Request - Use your Access Token to make a Post to the Batch API with the required parameters as defined below
Receive Batch ID - That API call kicks off the generation of the batch file on Black Diamond’s platform and returns a batch ID in the response to your call.
Monitor & Retrieve Batch Data - Monitor the status of the batch and use that batch ID to retrieve the batch file when ready.
Workflow in Detail
Authenticate User
Authenticate your user using the Auth Code flow defined here or a previously acquired refresh token.
Initiate Batch Request
Using the acquired access token, make a https POST call to the batch end point. *V5 URL Coming Soon*
Reminder, all request headers must always include:
Authorization: Bearer {{AccessToken}}
OCP-APIM-Subscription-Key: {{SubscriptionKeyFromPortal}}
Content-Type: application/json
Required Parameter:
ReturnDate - The As of Date that the data file will be generated for
batchFiles - provides the ability to limit which files are received in the batch to reduce the batch size if all files are not needed. **By default, no files are generated unless specifically requested.**
Accepted values are: "Accounts", "Portfolio", "Relationship", "Client", "Target", "Goals, Contact, FirmUser".
Note: if specifying "Portfolio", this will also bring back the associated "PortfolioGroup" json file.
Parameter Format:
"batchFiles":["Accounts", "Relationship"]
Optional Parameters:
By default, optional parameters are not included and will need to be opted into/requested in order for the information to be generated.
IncludeClosedAccounts - A true/false value that determines if accounts currently marked as closed on Black Diamond are included in the returned results. If not specified, defaults to false.
IncludeRecentlyClosedAccounts - includes closed accounts within the last 7 days based on the “returnDate” passed. *ignored if “IncludeClodedAccounts” = “true". If not specified, defaults to false.
IncludeCostBasis - A true/false value to determine if you want to include account level cost basis and tax lot information.
IncludeRecentlyClosedLots - show tax lots closed within 30 days of requested ”returnDate.” If not specified, defaults to NO closed lots in batch. *ignored if “includeCostBasis” = “false”
CostBasisByHolding - A true/false to break down cost basis at a holding level.
IncludeTransactions - A true/false value that determines if account level transactions are returned. Default returns last 7 days of transactions based up on the specified “returnDate”.
SingleDayTransactions - returns only the transactions on the specified “returnDate”. *ignored if “includeTransactions” = “false”
TransactionDayLimit - determines how many calendar days to look back to pull in transactions based upon the specified “returnDate”. Accepted values are [1-31] and “MTD” (month to date). *ignored if “includeTransactions” = “false”
IncludeHoldings - A true/false value that determines if account level holdings are returned. If not specified, defaults to false.
IncludeAllocation - A true/false value that determines if account class and segment allocation are returned. If not specified, defaults to false.
TargetPointsOfPrecision - 1-6, defaults to 4 and directly impacts ToleranceLower and ToleranceUpper
resultCallbackUrl - the batch API provides the ability to be notified via a webhook when the Batch file is complete and ready to be downloaded. The value provided in this parameter should be the full URL for the API to call back once your batch has completed. See below section on Accessing Batch Data for more information
clientToken - *Recommended if using a Callback URL* the client token is a value that should be generated per call and should be unique. BlackDiamond will store this token and send it to the provided callback URL; this helps ensure the call to your URL is a call you expected and not some arbitrary call. We recommend you compare this value to ensure the call to your callback end point is valid.
Optional Batch By Data Format Parameters:
These parameters are often used in conjunction with the firm status endpoint. More information can be found under later in the article under the "How do I call the firm status call?" section.
DataProviderIds - A true/false value to filter the account.json files to just the specified data formats/providers. If not specified, defaults to all data formats.
Format: DataProviderIds: [123, 456, 12340]
IsReconciled - A true/false value to only include accounts that have completed reconciliation. If used in conjunction with the DataProviderIds parameter then only the specified data formats that have completed reconciliation will generate account information. If not provided, default is false. Additionally, closed accounts would be excluded.
Example Batch Request Body
{
"returnDate": "01-01-2024",
"batchFiles": ["Accounts"],
"includeCostBasis": true,
"includeTransactions": true,
"TransactionDayLimit": 31,
"includeHoldings": true,
"includeAllocation": true,
}
This above example would return similar information to our previous batch versions, but please modify with additional batch files or parameters to fit your use case. Highlighted fields are required, a minimum of one batch file type will need to be provided.
Receive Batch ID
If a successful post is made, the API with respond with the id for your specific batch request. Sample POST response from /batch:
{
"id": "826217bd-b0c7-40f6-a1db-5c3e4229399b"
}
Monitor & Retrieve Batch Data
Once a batch POST has been initiated, the time to generate the file will vary by firm depending on number of accounts. The typical file generation time is a few minutes but can run up to an hour for very large firms. There are two ways to retrieve the batch data when ready.
Utilize a Webhook Notification
Poll the Batch Endpoint
When a successful call is made to the GET /batch/{BatchId} the batch zip file will be streamed to you.
Webhook Notification
Black Diamond supports the concept of a “webhook” or “call back” URL. This means that once the batch file completes and is ready for download, Black Diamond will notify you of completion via the provided URL. A subsequent call is required to download the file, but the webhook eliminates the need to poll the GET /batch/{BatchId} endpoint until completion.
If a callback URL is provided as a parameter in the Batch post, Black Diamond will make a post to that URL when the batch file is finished generating with the below structure:
{
"batchRef": "826217bd-b0c7-40f6-a1db-5c3e4229399b",
"fileID": 70258143,
"returnDate": "2021-12-31",
"includeClosedAccounts": true,
"status": "Finished",
"callBackUri": "http://www.yourcallbackurlhere/",
"clientToken": "SomeUniqueValue",
"responseType": "Json"
}
When that notification is received you would make a GET call to /batch/{BatchId} to receive the zip file.
Poll Batch Endpoint
If you do not want to utilize the webhook notification, you can poll the GET call at /batch/{BatchId} to monitor the status of the batch file. The typical file generate time is a few minutes but can run up to an hour for very large firms. The response codes you may receive when polling this end point are detailed below.
Frequently Asked Questions
When will the data be ready each morning to be pulled?
Black Diamond is a T-1 system and thus receives account data files overnight from our data sources and reconciles that information each morning. We are dependent on the timing of receiving that data each morning and so when the data is ready for each firm will vary and is dependent on which data sources or custodians they utilize.
However we have a commitment to have all accounts through automated reconciliation by 9:00AM EST and on average most firms’ data becomes available between 7 and 8 am EST. On a daily basis, about 99% of accounts clear through automated reconciliation and about 1% of accounts are held out as exceptions for manual review. We do not have a specific time to clear all exceptions in a day. (generally it is clear between 9-11am ET and is always done by the end of the day)
We strongly recommend you utilize our Firm Status get call to monitor when a user or firms data has been updated through the previous business day.
***As of September 2023, the Black Diamond reconciliation process was enhanced to allow for accounts to be updated(T-1) on a data format level rather than a firm level. Previously, account as of dates would not be updated until recon for all custodians at the firm had finalized, but this enhancement will now update the account as of dates as soon as that particular custodian's recon is completed allowing you to potentially get data even earlier. The Firm Status endpoint has historically been used to check the recon status at a firm level, but it can be utilized to look at an individual custodian's recon completion time as well.
How do I call the firm status call?
The firm status call can be accessed via the API section of this site under Firm API --> POST Firm Recon Status V2. Here are the different status responses you can expect:
Not Started – Black Diamond’s automated reconciliation process has not begun.
In Progress – Some but not all data providers have been processed through the automated reconciliation engine. Once all data providers have been competed the firm status will update from In Progress to Complete.
Complete (Exceptions) – **This is the status when the data becomes available for users to see on the Black Diamond UI** All available data providers have been completed through Black Diamond’s automated reconciliation engine. Exception is displayed due to an account not passing Black Diamond’s automated audits and now needs attention from a member of the operations team.
Complete (Aggregator) – All non-aggregated data providers have been completed through the Black Diamond automated reconciliation engine and no exceptions exists. The following providers are considered aggregated data providers and typically arrive later in the day (Plais, ByAll Accounts).
Completed - All available data providers have been completed through Black Diamond’s automated reconciliation engine and there are no exceptions.
Sample response to the Firm Status V2 call:
{
"asOfDate": "2024-02-29T00:00:00Z",
"finalizeCompleteTime": "2024-03-01T12:30:43.933Z",
"firmReconStatus": "Complete (Aggregator)",
"totalExceptions": 0,
"dataProviderDetails": [{
"dataProviderName": "ByAllAccounts",
"dataProviderId": 156,
"exceptions": [],
"lastCompletedRGLDate": null,
"lastCompletedUGLDate": null,
"reconciliationCompleteTime": null,
"requiredForFinalize": false,
"status": "Not Started",
"totalExceptions": 0
}, {
"dataProviderName": "Vanguard - Flagship ADF",
"dataProviderId": 1221,
"exceptions": [],
"lastCompletedRGLDate": null,
"lastCompletedUGLDate": null,
"reconciliationCompleteTime": "2024-03-01T02:04:49Z",
"requiredForFinalize": true,
"status": "Complete",
"totalExceptions": 0
}, {
"dataProviderName": "Schwab",
"dataProviderId": 551,
"exceptions": [],
"lastCompletedRGLDate": "2024-02-29T00:00:00Z",
"lastCompletedUGLDate": "2024-02-29T00:00:00Z",
"reconciliationCompleteTime": "2024-03-01T09:03:22Z",
"requiredForFinalize": true,
"status": "Complete",
"totalExceptions": 0
}, {
"dataProviderName": "Fidelity IWS",
"dataProviderId": 1247,
"exceptions": [],
"lastCompletedRGLDate": null,
"lastCompletedUGLDate": null,
"reconciliationCompleteTime": "2024-03-01T09:03:34Z",
"requiredForFinalize": true,
"status": "Complete",
"totalExceptions": 0
}, {
"dataProviderName": "Fidelity CGF",
"dataProviderId": 196,
"exceptions": [],
"lastCompletedRGLDate": null,
"lastCompletedUGLDate": null,
"reconciliationCompleteTime": "2024-03-01T09:46:12Z",
"requiredForFinalize": true,
"status": "Complete",
"totalExceptions": 0
}
]}
What are the response codes when trying to download a batch file?
Response Codes:
200: File is ready and downloading.
202: You have submitted a batch that is currently in progress. Please check back later.
401: Token is expired or the user does not have access to retrieve the data you are seeking anymore
403: Missing terms and conditions. This user has to log in and accept the T&C.
404: Standard not found implementation. Following points of clarification:
Batch is purged or currently unavailable.
404 is also returned when the path or values are incorrect, missing, or not found.
429: Rate limited. Too many requests in a given time period. The rate limits will be set appropriately for you given that you’ll be submitting a batch per user. If running into this consistently please reach out to your Black Diamond contact.
500: This is an unexpected error. It should return a correlation ID along with the error. Providing this correlation ID to Black Diamond helps us troubleshoot the concern.