Deserialize hits memory limit
We have a custom apex webservice in which we do certain operations and later call a managed package apex method that returns a JSON string which we deserialize and send back to the client. For example purposes let's assume the following JSON structure is returned from the managed package method:
{
"x": "xxx",
"y": "yyy",
"z": "zzz",
"lines": [{
"a": "aaa",
"b": "bbb",
"c": "ccc",
"record": {
"attributes": {
"type": "Account",
"url": "/services/data/v44.0/sobjects/Account/001000000000000000"
},
"Id": "001000000000000000",
"AccountNumber": "12345",
"Contacts": {
"totalSize": 1,
"done": true,
"records": [{
"attributes": {
"type": "Contact",
"url": "/services/data/v44.0/sobjects/Contact/003S00000000000000"
},
"Id": "003S00000000000000",
"Phone": "22323423423",
"MobilePhone": "345345345345"
}]
}
}
}]
}
Taking the above example we could assume that we could deserialize the above JSON string into the following classes:
public class MainModel {
String x;
String y;
String z;
LinesModel lines;
}
public class LineModel {
String a;
String b;
String c;
Account record;
}
As we can see the LineModel
states that the record
attribute has an Account
sobject type, therefore it can span children relationships of Contacts, Opportunities and many other children relationships.
Therefore we could deserialize the JSON the following way:
MainModel model = (MainModel)JSON.deserialize(jsonString, MainModel.class);
Before returning the model to our webservice caller we want to get rid of some information that the manage package includes but we don't want, for example the Contacts
children tree for each Account
.
In order to remove it, we are doing the following:
Map<String, Object> genericModel = (Map<String, Object>)JSON.deserializeUntyped(jsonString);
Object lines = (Object)genericModel.get('lines');
Map<String, Object> itemObj;
Map<String, Object> record;
for (Object item : lines) {
itemObj = (Map<String, Object>)item;
record = (Map<String, Object>)itemObj.get('record');
if (record.containsKey('Contacts')) {
record.remove('Contacts')
}
}
MainModel model = (MainModel)JSON.deserialize(JSON.serialize(genericModel), MainModel.class);
The issue we're facing is that when the response from the manage package has many nested records the desarialization process (the deserializeUntyped
call) fails with an Apex heap memory limit. When I say nested records, in the context of the above example there could maybe be only 20 Accounts but each account maybe having 60 contacts.
Since we don't have control on what the manage package returns in the JSON we cannot avoid having those unnecessary children.
We've tried removing the children using regex directly on the JSON string the following way to avoid unnecessary deserialization for cleaning:
String textPattern = '\"Contacts\":.*records.*\[.*]},';
String resultText = jsonString.replaceAll(textPattern, '');
MainModel model = (MainModel)JSON.deserialize(JSON.serialize(resultText), MainModel.class);
But this results on a System.LimitException: Regex too complicated error because the JSON string is too large.
Any ideas on how we can clean the unwanted JSON data before returning it to the caller that could be viable?
Please take into account that the above is an example very simplified for the sake of understanding the real issue, the real JSON has many other relationships and objects that need cleaning and is more complex but the scenario is pretty much the same.
json deserialize heap
add a comment |
We have a custom apex webservice in which we do certain operations and later call a managed package apex method that returns a JSON string which we deserialize and send back to the client. For example purposes let's assume the following JSON structure is returned from the managed package method:
{
"x": "xxx",
"y": "yyy",
"z": "zzz",
"lines": [{
"a": "aaa",
"b": "bbb",
"c": "ccc",
"record": {
"attributes": {
"type": "Account",
"url": "/services/data/v44.0/sobjects/Account/001000000000000000"
},
"Id": "001000000000000000",
"AccountNumber": "12345",
"Contacts": {
"totalSize": 1,
"done": true,
"records": [{
"attributes": {
"type": "Contact",
"url": "/services/data/v44.0/sobjects/Contact/003S00000000000000"
},
"Id": "003S00000000000000",
"Phone": "22323423423",
"MobilePhone": "345345345345"
}]
}
}
}]
}
Taking the above example we could assume that we could deserialize the above JSON string into the following classes:
public class MainModel {
String x;
String y;
String z;
LinesModel lines;
}
public class LineModel {
String a;
String b;
String c;
Account record;
}
As we can see the LineModel
states that the record
attribute has an Account
sobject type, therefore it can span children relationships of Contacts, Opportunities and many other children relationships.
Therefore we could deserialize the JSON the following way:
MainModel model = (MainModel)JSON.deserialize(jsonString, MainModel.class);
Before returning the model to our webservice caller we want to get rid of some information that the manage package includes but we don't want, for example the Contacts
children tree for each Account
.
In order to remove it, we are doing the following:
Map<String, Object> genericModel = (Map<String, Object>)JSON.deserializeUntyped(jsonString);
Object lines = (Object)genericModel.get('lines');
Map<String, Object> itemObj;
Map<String, Object> record;
for (Object item : lines) {
itemObj = (Map<String, Object>)item;
record = (Map<String, Object>)itemObj.get('record');
if (record.containsKey('Contacts')) {
record.remove('Contacts')
}
}
MainModel model = (MainModel)JSON.deserialize(JSON.serialize(genericModel), MainModel.class);
The issue we're facing is that when the response from the manage package has many nested records the desarialization process (the deserializeUntyped
call) fails with an Apex heap memory limit. When I say nested records, in the context of the above example there could maybe be only 20 Accounts but each account maybe having 60 contacts.
Since we don't have control on what the manage package returns in the JSON we cannot avoid having those unnecessary children.
We've tried removing the children using regex directly on the JSON string the following way to avoid unnecessary deserialization for cleaning:
String textPattern = '\"Contacts\":.*records.*\[.*]},';
String resultText = jsonString.replaceAll(textPattern, '');
MainModel model = (MainModel)JSON.deserialize(JSON.serialize(resultText), MainModel.class);
But this results on a System.LimitException: Regex too complicated error because the JSON string is too large.
Any ideas on how we can clean the unwanted JSON data before returning it to the caller that could be viable?
Please take into account that the above is an example very simplified for the sake of understanding the real issue, the real JSON has many other relationships and objects that need cleaning and is more complex but the scenario is pretty much the same.
json deserialize heap
5
This doesn't sound like a memory leak to me, just a very large json string. The synchronous limit on heap size is ~6MB, and the async limit is ~12MB. If you're trying to manipulate that JSON, I'd expect that you'd only be able to handle a JSON string less than half that size (the string remains in memory, and you need at least as much memory to deserialize). Do you happen to have any idea of the size of your JSON string(s)?
– Derek F
3 hours ago
1
I've edited the question title to reflect Derek's point that this isn't a leak but rather hitting the limit.
– Keith C
58 mins ago
add a comment |
We have a custom apex webservice in which we do certain operations and later call a managed package apex method that returns a JSON string which we deserialize and send back to the client. For example purposes let's assume the following JSON structure is returned from the managed package method:
{
"x": "xxx",
"y": "yyy",
"z": "zzz",
"lines": [{
"a": "aaa",
"b": "bbb",
"c": "ccc",
"record": {
"attributes": {
"type": "Account",
"url": "/services/data/v44.0/sobjects/Account/001000000000000000"
},
"Id": "001000000000000000",
"AccountNumber": "12345",
"Contacts": {
"totalSize": 1,
"done": true,
"records": [{
"attributes": {
"type": "Contact",
"url": "/services/data/v44.0/sobjects/Contact/003S00000000000000"
},
"Id": "003S00000000000000",
"Phone": "22323423423",
"MobilePhone": "345345345345"
}]
}
}
}]
}
Taking the above example we could assume that we could deserialize the above JSON string into the following classes:
public class MainModel {
String x;
String y;
String z;
LinesModel lines;
}
public class LineModel {
String a;
String b;
String c;
Account record;
}
As we can see the LineModel
states that the record
attribute has an Account
sobject type, therefore it can span children relationships of Contacts, Opportunities and many other children relationships.
Therefore we could deserialize the JSON the following way:
MainModel model = (MainModel)JSON.deserialize(jsonString, MainModel.class);
Before returning the model to our webservice caller we want to get rid of some information that the manage package includes but we don't want, for example the Contacts
children tree for each Account
.
In order to remove it, we are doing the following:
Map<String, Object> genericModel = (Map<String, Object>)JSON.deserializeUntyped(jsonString);
Object lines = (Object)genericModel.get('lines');
Map<String, Object> itemObj;
Map<String, Object> record;
for (Object item : lines) {
itemObj = (Map<String, Object>)item;
record = (Map<String, Object>)itemObj.get('record');
if (record.containsKey('Contacts')) {
record.remove('Contacts')
}
}
MainModel model = (MainModel)JSON.deserialize(JSON.serialize(genericModel), MainModel.class);
The issue we're facing is that when the response from the manage package has many nested records the desarialization process (the deserializeUntyped
call) fails with an Apex heap memory limit. When I say nested records, in the context of the above example there could maybe be only 20 Accounts but each account maybe having 60 contacts.
Since we don't have control on what the manage package returns in the JSON we cannot avoid having those unnecessary children.
We've tried removing the children using regex directly on the JSON string the following way to avoid unnecessary deserialization for cleaning:
String textPattern = '\"Contacts\":.*records.*\[.*]},';
String resultText = jsonString.replaceAll(textPattern, '');
MainModel model = (MainModel)JSON.deserialize(JSON.serialize(resultText), MainModel.class);
But this results on a System.LimitException: Regex too complicated error because the JSON string is too large.
Any ideas on how we can clean the unwanted JSON data before returning it to the caller that could be viable?
Please take into account that the above is an example very simplified for the sake of understanding the real issue, the real JSON has many other relationships and objects that need cleaning and is more complex but the scenario is pretty much the same.
json deserialize heap
We have a custom apex webservice in which we do certain operations and later call a managed package apex method that returns a JSON string which we deserialize and send back to the client. For example purposes let's assume the following JSON structure is returned from the managed package method:
{
"x": "xxx",
"y": "yyy",
"z": "zzz",
"lines": [{
"a": "aaa",
"b": "bbb",
"c": "ccc",
"record": {
"attributes": {
"type": "Account",
"url": "/services/data/v44.0/sobjects/Account/001000000000000000"
},
"Id": "001000000000000000",
"AccountNumber": "12345",
"Contacts": {
"totalSize": 1,
"done": true,
"records": [{
"attributes": {
"type": "Contact",
"url": "/services/data/v44.0/sobjects/Contact/003S00000000000000"
},
"Id": "003S00000000000000",
"Phone": "22323423423",
"MobilePhone": "345345345345"
}]
}
}
}]
}
Taking the above example we could assume that we could deserialize the above JSON string into the following classes:
public class MainModel {
String x;
String y;
String z;
LinesModel lines;
}
public class LineModel {
String a;
String b;
String c;
Account record;
}
As we can see the LineModel
states that the record
attribute has an Account
sobject type, therefore it can span children relationships of Contacts, Opportunities and many other children relationships.
Therefore we could deserialize the JSON the following way:
MainModel model = (MainModel)JSON.deserialize(jsonString, MainModel.class);
Before returning the model to our webservice caller we want to get rid of some information that the manage package includes but we don't want, for example the Contacts
children tree for each Account
.
In order to remove it, we are doing the following:
Map<String, Object> genericModel = (Map<String, Object>)JSON.deserializeUntyped(jsonString);
Object lines = (Object)genericModel.get('lines');
Map<String, Object> itemObj;
Map<String, Object> record;
for (Object item : lines) {
itemObj = (Map<String, Object>)item;
record = (Map<String, Object>)itemObj.get('record');
if (record.containsKey('Contacts')) {
record.remove('Contacts')
}
}
MainModel model = (MainModel)JSON.deserialize(JSON.serialize(genericModel), MainModel.class);
The issue we're facing is that when the response from the manage package has many nested records the desarialization process (the deserializeUntyped
call) fails with an Apex heap memory limit. When I say nested records, in the context of the above example there could maybe be only 20 Accounts but each account maybe having 60 contacts.
Since we don't have control on what the manage package returns in the JSON we cannot avoid having those unnecessary children.
We've tried removing the children using regex directly on the JSON string the following way to avoid unnecessary deserialization for cleaning:
String textPattern = '\"Contacts\":.*records.*\[.*]},';
String resultText = jsonString.replaceAll(textPattern, '');
MainModel model = (MainModel)JSON.deserialize(JSON.serialize(resultText), MainModel.class);
But this results on a System.LimitException: Regex too complicated error because the JSON string is too large.
Any ideas on how we can clean the unwanted JSON data before returning it to the caller that could be viable?
Please take into account that the above is an example very simplified for the sake of understanding the real issue, the real JSON has many other relationships and objects that need cleaning and is more complex but the scenario is pretty much the same.
json deserialize heap
json deserialize heap
edited 1 hour ago
Keith C
94.6k1089204
94.6k1089204
asked 4 hours ago
jonathanwieseljonathanwiesel
252114
252114
5
This doesn't sound like a memory leak to me, just a very large json string. The synchronous limit on heap size is ~6MB, and the async limit is ~12MB. If you're trying to manipulate that JSON, I'd expect that you'd only be able to handle a JSON string less than half that size (the string remains in memory, and you need at least as much memory to deserialize). Do you happen to have any idea of the size of your JSON string(s)?
– Derek F
3 hours ago
1
I've edited the question title to reflect Derek's point that this isn't a leak but rather hitting the limit.
– Keith C
58 mins ago
add a comment |
5
This doesn't sound like a memory leak to me, just a very large json string. The synchronous limit on heap size is ~6MB, and the async limit is ~12MB. If you're trying to manipulate that JSON, I'd expect that you'd only be able to handle a JSON string less than half that size (the string remains in memory, and you need at least as much memory to deserialize). Do you happen to have any idea of the size of your JSON string(s)?
– Derek F
3 hours ago
1
I've edited the question title to reflect Derek's point that this isn't a leak but rather hitting the limit.
– Keith C
58 mins ago
5
5
This doesn't sound like a memory leak to me, just a very large json string. The synchronous limit on heap size is ~6MB, and the async limit is ~12MB. If you're trying to manipulate that JSON, I'd expect that you'd only be able to handle a JSON string less than half that size (the string remains in memory, and you need at least as much memory to deserialize). Do you happen to have any idea of the size of your JSON string(s)?
– Derek F
3 hours ago
This doesn't sound like a memory leak to me, just a very large json string. The synchronous limit on heap size is ~6MB, and the async limit is ~12MB. If you're trying to manipulate that JSON, I'd expect that you'd only be able to handle a JSON string less than half that size (the string remains in memory, and you need at least as much memory to deserialize). Do you happen to have any idea of the size of your JSON string(s)?
– Derek F
3 hours ago
1
1
I've edited the question title to reflect Derek's point that this isn't a leak but rather hitting the limit.
– Keith C
58 mins ago
I've edited the question title to reflect Derek's point that this isn't a leak but rather hitting the limit.
– Keith C
58 mins ago
add a comment |
1 Answer
1
active
oldest
votes
In your wrapper class LineModel
, don't use Standard Account. If you do that you will also deserialize children. Instead, create Apex Wrapper for Account with fields you need. This will be more memory efficient.
public class LineModel {
String a;
String b;
String c;
MyAccount record;
}
public Class MyAccount{
String Id;
String AccountNumber;
}
public class MainModel {
String x;
String y;
String z;
List<LineModel> lines;
}
+1. This is indeed a very neat approach to deal with any such limitations.
– Jayant Das
8 mins ago
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "459"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsalesforce.stackexchange.com%2fquestions%2f246040%2fdeserialize-hits-memory-limit%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
In your wrapper class LineModel
, don't use Standard Account. If you do that you will also deserialize children. Instead, create Apex Wrapper for Account with fields you need. This will be more memory efficient.
public class LineModel {
String a;
String b;
String c;
MyAccount record;
}
public Class MyAccount{
String Id;
String AccountNumber;
}
public class MainModel {
String x;
String y;
String z;
List<LineModel> lines;
}
+1. This is indeed a very neat approach to deal with any such limitations.
– Jayant Das
8 mins ago
add a comment |
In your wrapper class LineModel
, don't use Standard Account. If you do that you will also deserialize children. Instead, create Apex Wrapper for Account with fields you need. This will be more memory efficient.
public class LineModel {
String a;
String b;
String c;
MyAccount record;
}
public Class MyAccount{
String Id;
String AccountNumber;
}
public class MainModel {
String x;
String y;
String z;
List<LineModel> lines;
}
+1. This is indeed a very neat approach to deal with any such limitations.
– Jayant Das
8 mins ago
add a comment |
In your wrapper class LineModel
, don't use Standard Account. If you do that you will also deserialize children. Instead, create Apex Wrapper for Account with fields you need. This will be more memory efficient.
public class LineModel {
String a;
String b;
String c;
MyAccount record;
}
public Class MyAccount{
String Id;
String AccountNumber;
}
public class MainModel {
String x;
String y;
String z;
List<LineModel> lines;
}
In your wrapper class LineModel
, don't use Standard Account. If you do that you will also deserialize children. Instead, create Apex Wrapper for Account with fields you need. This will be more memory efficient.
public class LineModel {
String a;
String b;
String c;
MyAccount record;
}
public Class MyAccount{
String Id;
String AccountNumber;
}
public class MainModel {
String x;
String y;
String z;
List<LineModel> lines;
}
edited 6 mins ago
Jayant Das
12.5k2723
12.5k2723
answered 3 hours ago
Pranay JaiswalPranay Jaiswal
14k32351
14k32351
+1. This is indeed a very neat approach to deal with any such limitations.
– Jayant Das
8 mins ago
add a comment |
+1. This is indeed a very neat approach to deal with any such limitations.
– Jayant Das
8 mins ago
+1. This is indeed a very neat approach to deal with any such limitations.
– Jayant Das
8 mins ago
+1. This is indeed a very neat approach to deal with any such limitations.
– Jayant Das
8 mins ago
add a comment |
Thanks for contributing an answer to Salesforce Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsalesforce.stackexchange.com%2fquestions%2f246040%2fdeserialize-hits-memory-limit%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
5
This doesn't sound like a memory leak to me, just a very large json string. The synchronous limit on heap size is ~6MB, and the async limit is ~12MB. If you're trying to manipulate that JSON, I'd expect that you'd only be able to handle a JSON string less than half that size (the string remains in memory, and you need at least as much memory to deserialize). Do you happen to have any idea of the size of your JSON string(s)?
– Derek F
3 hours ago
1
I've edited the question title to reflect Derek's point that this isn't a leak but rather hitting the limit.
– Keith C
58 mins ago