How Big is TOO BIG for JSON?

39 Comments

1. 

   jsoh zeigler on November 14, 2014 at 4:30 pm

   thanks! that was useful info.

   Reply
   • 

     Json on December 10, 2014 at 6:44 am

     In my tests IE9 fail with JSON responses above 40-45MB

     Reply
     • 

       Mr. JSON on March 10, 2015 at 6:29 am

       It might have something to do with the MaxJsonLength property.

       Reply
   • 

     pravEENA on January 10, 2017 at 7:29 pm

     Why is browser using 50MB RAM to load 15MB file?

     100,000 (15.5MB) 2:00 DOM Ready: 1.24s
     RAM: 150.4MB
     Loading: usable
     Loaded: usable DOM Ready: 1.62s
     RAM: 424.0MB
     Loading: usable
     Loaded: usable DOM Ready: 463ms
     RAM: 296.2MB
     Loading: untestable
     Loaded: usable DOM Ready: 1.71s
     RAM: 210.6MB
     Loading: untestable
     Loaded: usable

     Reply
     • 

       Josh Zeigler on January 12, 2017 at 10:47 am

       This difference includes general application overhead. That memory is not specific to the loading of the 15MB file, but all browser processes.

       Reply
     • 

       BLAH BLAH on April 20, 2017 at 7:41 pm

       The JSON file (which is essentially just text) is 15MB, but the browser has to use javascript to parse it all into data and then run queries against. Depending on how the javascript is written, there can be a lot of variables and arrays generated in the back-end sitting in RAM (eg: a lot of JSON data will use arrays for sub-sets of data, eg: like a person with address array data.) these arrays can get huge if they’re created and then used to write the html rather then just writing the html line by line directly.

       Reply
2. 

   Ben on May 26, 2015 at 9:05 pm

   It was really a good article that I have been looking for. The problem is we don’t need to send such a huge object to a browser, unless the user initially requested it.

   Most applicable application for sending large JSON objects to browser is blog posts (Dynamically loaded to the DOM) which doesn’t necessary means we have to pull 100 long posts at once.

   You can easily destroy the object after manipulating the DOM. This was the browser’s memory will become free for the next batch.

   Thanks

   Reply
3. 

   Kat on June 5, 2015 at 10:23 pm

   Thank you for the article. I was wondering if there are any limits on the size of the text in JSON value pairs? For example, if we have a JSON object for “Games” and one of the values is “Game rules” that has long text, is there a limit of the size of that text?

   Example:
   “game” : {
   “game_name”: “Some Game”,
   “publisher”: “Some publisher”,
   “game_rules”: “very long text here….”;
   “game_status” : “in development”
   }

   Reply
   • 

     Josh Zeigler on June 6, 2015 at 8:46 am

     JavaScript has no formal standard maxlength for a String value. This maximum is determined by each individual implementation. I have read that IE’s maximum String size is 2^31 and that Mozilla supports closer to 2^15. But the real question is how long will it take to parse that large amount of data and is it worth it to stick with a data-driven JSON approach? It seems to me that this data might be best returned to the client application in pages or consumable chunks versus instead. It might even make sense to return this as HTML or HTML fragments depending on the implementation.

     Reply
     • 

       TJ Colgan on September 13, 2017 at 2:44 pm

       If I have this single json object:
       “test : {
       “variable”: “a very long long string”
       }

       How much overhead do you think in serialization/deserialization/download over http if i keep increasing that long string? I’m trying to see if there is performance issue if I put a super big string as a value

       Reply
       • 

         Josh Zeigler on September 13, 2017 at 5:21 pm

         There is no formal string length maximum in JavaScript. So this is left to each individual implementation. However, in theory, this can be very large, upwards of hundreds of megabytes. For Example, Chrome can support up to 512MB for an allocation request. Likely, most browsers will run out of memory before being able to allocate this much to a single member. In a more practical view, “super big” and “very long long” are relative and impossible to determine if it will cause an issue in your case. How large are the strings you are planning to work with?

         Reply
4. 

   Rune Jeppesen on July 1, 2015 at 8:31 am

   Thanks!
   Have you tried doing the same test recently?
   And in the new Microsoft browser Edge/Spartan?

   Reply
   • 

     Josh Zeigler on July 5, 2015 at 9:25 am

     No, unfortunately I haven’t had a chance to re-run this test. Good call though and after I finish up the project I am on, I will try and find some time to do that.

     Reply
5. 

   Hamlin on August 20, 2015 at 11:26 am

   Thanks for the information, very useful. You say 10,000 records is your sweet spot. However, in looking at the download time of 12 seconds, I am wondering how you can say that’s a good sweet spot. Doesn’t that seem to be unreasonably long to download data? Or maybe I am missing something? Thanks

   Reply
   • 

     Josh Zeigler on August 20, 2015 at 1:52 pm

     In perspective to many of the business web applications I work on, this 12 second “sweet spot” is very acceptable. I wouldn’t recommend this “sweet spot” for a web site or shopping experience and would expect this to be much lower there.

     Reply
6. 

   Havia on October 28, 2015 at 10:25 am

   I planned to try a big json data (100MB) in plain javacript, and most of my website visitor will be using dial up connection. What method would be useful to help it work faster. async ??? Plese reply. Thank you.

   Reply
   • 

     Josh Zeigler on November 6, 2015 at 9:42 am

     I’m not sure that much of anything will help user’s retrieve a 100MB file over dial-up. In theory, asynchronous communication could technically slow down the response in this scenario if other async resources were loading. If my customers were using dial-up or any slower data connections, I would also assume that their devices don’t have a lot of resources or modern support. I would stick with classic server-side rendering and work to make the data as small as possible.

     Reply
     • 

       Havia on November 18, 2015 at 5:23 am

       Thank you for your quick response. So, think I should leave this idea behind, and will go for server side.

       Reply
7. 

   David on January 4, 2016 at 11:31 am

   Thanks, very useful post 🙂

   Reply
8. 

   TB on May 12, 2016 at 9:57 pm

   Thanks for giving this information. Also I have tested this on node.js, I thought that it was the limitation on V8 only, but after reading your blog, it seems that all browsers had this limitation. So I assume that all browser’s javascript engine including V8 on node.js cannot handle too large json & object.

   Test: https://git.io/vrYmr

   Reply
9. 

   Greg on August 9, 2016 at 11:25 am

   Josh, have you had a chance to re-run that test in IE11?

   Reply
10. 

    Ranjith on October 21, 2016 at 10:09 am

    Thanks, I was wondering if I can use it for 1000 records or not!

    Reply
11. 

    Harry on July 10, 2017 at 6:10 pm

    I have a 200 GB JSON file, is there any possibility can be processed from an Application.

    Reply
    • 

      Josh Zeigler on July 17, 2017 at 4:39 pm

      I don’t know anything about the Application you are speaking of. Assuming it is a web app and you wanted to process the 200 GB JSON file client-side, there are a variety of reasons this may not be a good practice. Even if the browser could handle this amount of data, which I highly doubt even modern browsers could, the first challenge would be the time it would take for the user to download that file. They certainly wouldn’t be processing this in real time. Another challenge would be data allowance concerns. It is becoming increasingly common for user’s to be on limited data connections. I would suggest running this data through a database or some other tool and returning only the necessary fragments of data to the user. Again, I have no context to your application here, but in general this would not be advisable.

      Reply
12. 

    Dan on October 8, 2017 at 9:17 pm

    Interesting article. Do you think a transformation like this might make it better in general?
    So pass the same data as multiple large rows of primitives instead of one large array of objects:

    var testData =
    {
    “ACCTOUNT_NUMBERS”: [“1234567890”,…],
    “CUSTOMER_NAMES”: [“ACME Products and Services, Inc.”,…],
    “ADDRESSES”: [“123 Main Street”,…],
    “CITIES”:[“Albuquerque”,…],
    “STATES”:[“NM”,…],
    “ZIPS”:[“87101-1234”,…]
    }

    Reply
    • 

      Josh Zeigler on October 9, 2017 at 9:07 am

      That is a great thought and could certainly reduce the overall size of the object. It would be interesting to see the performance in that model. The challenging part would be maintaining the position of all of the arrays so that the individual records are maintained.

      Reply
      • 

        gertrude on September 19, 2018 at 5:01 pm

        Also worth pointing out, just in terms of transfer time/size using a data structure like that, if your application is already using GZIP compression from the server, you wouldn’t get much if any additional size reduction of the payload.

        Reply
13. 

    Chitra on January 7, 2018 at 7:23 am

    Hi,
    We maintain long json file containing all the datasets iteration logic and it gets checked into source code by multiple developers.
    Maintaining and validating this big json file becomes time consuming.

    We opted to put all angular and html transformation into json file, so that during runtime the file can be parsed seamlessly to get UI rendered with the given filterations.

    Can you provide any thoughts on separating json file into multiple and storing it or Is there any best way to maintain the big json file?

    Reply
    • 

      Josh Zeigler on January 12, 2018 at 9:24 am

      Large JSON files can be nearly impossible to manage manually as a flat file. I would suggest storing the data in another storage and management tool that exposes an API to serve the data or to export a static data file. This should simplify maintenance. I would also recommend trying to find ways to make the data smaller for your clients to download if possible. If users don’t need all of the data you are sending, an API that can return only the necessary data would be best.

      Reply
14. 

    moloth on November 8, 2018 at 5:14 am

    Hello, thank you for this measuring. Could you please tell me, if you know whether exists the same measuring for Android applications. I haven’t found in the Internet info about the largest JSON object they could receive and the time they need to process it.

    Reply
    • 

      Josh Zeigler on November 11, 2018 at 12:37 pm

      I haven’t done this evaluation on Android. I do plan on doing this again soon and plan to include mobile browsers and platforms as well.

      Reply
      • 

        Rodney on August 30, 2020 at 12:20 pm

        Hi Josh,

        I came across this excellent article while researching an issue one of my colleagues is having with a 50K+ record set. You mention in your last comment (2018) that you were planning to do this again, presumably with modern browsers (Chrome, Firefox, Edge, Safari). Are you still planning to do that? I would very much be interested to see that data.

        Thanks again. Very informative.

        Reply
15. 

    bob on November 11, 2018 at 11:33 am

    Great post! It was the #1 search result on Google for “correct way to structure json from large dataset.” I was wondering the most efficient way to build an object from a 1000+ records taken from an HTML table in Quickbase. I created a Chrome Extension that builds a single object from a paginated dataset with 200 records per page. Because of the CORS configuration, I’m sending the object in a post request to a proxy server where I built a NodeJS app that uses the Quickbase API to create a new table in a different Quickbase realm. I was debating sending a Json object from each page of Quickbase to the Node app, but I would rather pass the data through once instead of storing it on the proxy. I feel much more confident with the single object option now after reading your blog. Thanks again.

    Reply
16. 

    Ahmad on March 12, 2019 at 12:24 pm

    Great Article! Thank you

    Reply
17. 

    mono on April 29, 2019 at 3:18 am

    Great and useful job, thanks.

    Reply
18. 

    sergey on September 17, 2019 at 4:06 am

    To parse large JSON files, I use http://json-csv-xls.com. This is a bit off topic, but sometimes it helps.

    Reply
19. 

    Rami Ibrahim on February 20, 2020 at 11:16 am

    Thank you very much for this article.

    it really helped me decide whether to do client or server-side pagination.

    Thank you very much for your time and the efforts

    Reply
20. 

    Yuriy Novikov on February 25, 2020 at 12:05 pm

    Thank you for article.
    I worried that my json (600k) is too big for browser.
    You helped me to calm down.
    Really interesting how it going be on phones. From another side, modern phones can be more powerful than laptops.

    Reply
21. 

    Taher Ghulam Mohammed on April 8, 2020 at 5:38 pm

    Great post really informative.

    Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.