Reading Map objects from JSON array

[e-tobi]

The use case that drew my attention to json_events is, that I download a large JSON array where I want to process each object in the array one after the other. Loading the whole JSON into memory for parsing is a no-go. I'm new to Dart/Flutter (coming mainly from C#, C++, Python, Ruby) so I was little bit surprised there was no streaming JSON parser (I'm so used to JSON.Net, I'm probably a little bit spoiled :-) So: json_events to the rescue! (Thanks for providing this!)

But using the JsonArrayTraverser / JsonObjectTraverser feels a little "bulky". Wouldn't it be nice, if you could simply get Map<string, dynamic> objects from a JSON-Stream with an array? Then json_serializable can be used to get the objects from it.

I did a quick test, if this is possible - see code below. (That's where I stumbled upon the other issues, I just posted).
Wouldn't something like this be a nice addition to json_events? If you like this idea, I can try to provide a PR for this. The code below is not yet complete. e.g. an array of arrays wouldn't work. So this would need a little bit more time to do it right with unit tests etc..

readJsonObjectAsMap(StreamIterator<JsonEvent> streamIterator) async {
  Queue<dynamic> objectStack = Queue<dynamic>();
  Map<String, dynamic> root = {};
  objectStack.addLast(root);
  String? currentProperty;
  JsonEvent? lastEvent;
  while (objectStack.isNotEmpty) {
    await streamIterator.moveNext();
    print("${streamIterator.current.type} : ${streamIterator.current.value}");
    switch (streamIterator.current.type) {
      case JsonEventType.propertyName:
        currentProperty = streamIterator.current.value;
        break;
      case JsonEventType.propertyValue:
        if (currentProperty != null) {
          objectStack.last[currentProperty] = streamIterator.current.value;
          currentProperty = null;
        }
        break;
      case JsonEventType.beginObject:
        Map<String, dynamic> newObject = {};
        if (currentProperty != null) {
          objectStack.last[currentProperty] = newObject;
        }
        else {
          objectStack.last.add(newObject);
        }
        currentProperty = null;
        objectStack.addLast(newObject);
        break;
      case JsonEventType.endObject:
        objectStack.removeLast();
        break;
      case JsonEventType.beginArray:
        List newArray = [];
        objectStack.last[currentProperty] = newArray;
        currentProperty = null;
        objectStack.addLast(newArray);
        break;
      case JsonEventType.endArray:
        objectStack.removeLast();
        break;
      case JsonEventType.arrayElement:
        // There's a bug that null array element is emitted after an object in an array
        if (lastEvent?.type != JsonEventType.endObject)
        {
          objectStack.last.add(streamIterator.current.value);
        }
        break;
    }
    lastEvent = streamIterator.current;
  }
  return root;
}

[aeb-dev]

The use case that drew my attention to json_events is, that I download a large JSON array where I want to process each object in the array one after the other. Loading the whole JSON into memory for parsing is a no-go. I'm new to Dart/Flutter (coming mainly from C#, C++, Python, Ruby) so I was little bit surprised there was no streaming JSON parser (I'm so used to JSON.Net, I'm probably a little bit spoiled :-) So: json_events to the rescue! (Thanks for providing this!)

You're welcome! Hearing such things make the effort worth it.

But using the JsonArrayTraverser / JsonObjectTraverser feels a little "bulky". Wouldn't it be nice, if you could simply get Map<string, dynamic> objects from a JSON-Stream with an array? Then json_serializable can be used to get the objects from it.

I think this against the idea of json_events. Currently I don't accumulate anything because we can never be sure of the size. So if we were to return Map<> this means you are allocating whatever is coming through the stream which still can be GBs of data.

You can still accumulate yourself if you prefer, but I doubt this is what you want

class MyClass with JsonObjectTraverser {
  late int x;
  late List<MyClass> arr;
  late List<int> pArr;
  late List<List<int>> pNestedArray;
  String? text;

  Map<String, dynamic> jsonMap = {};

  @override
  Future<void> readJson(String key) async {
    switch (key) {
      case "x":
        jsonMap[key] = await this.readPropertyJsonContinue<int>();
        // rest of the code ...
    }
  }
}

[e-tobi]

You're right any sub-object may be large. In my case I know this isn't the case. I'm now using a JsonArrayToMapConverter so I can simply do: jsonEvenStream.transform(JsonArrayToMapConverter()) and get a Stream of (small) Map objects. these can then be mapped easily to real objects using json_serializable. Suites my needs. But I think this is a very common use case: Loading a list of data from a REST API or file with a large array but relative small elements.

[aeb-dev]

I believe the solution here is not providing a new converter but providing an easier way for people so they won't have to write too much code. I have been thinking about this for a while but I could not find a good way.

One way of doing this might be providing map instead of the readJson function. This map contains key, setter, an optional creator if the type is custom object.

class MyClass with JsonObjectTraverser {
  late int x;
  late List<MyClass> arr;

  Map<String, (void Function(dynamic) setter, dynamic Function()? creator)> fields => {
    "x": ((v) => x = v as int?, null),
    "arr": ((v) => arr.add(v as MyClass), MyClass.new),
  }
}

Don't mind the ugliness of the code 😄 We can improve that as well

[e-tobi]

Uhh... I started with Dart a week ago - this does not look easier to me. I still prefer my solution :-)

final response = await http.Client().send(
      http.Request("GET", Uri.parse("https://server.com/products")));

var productsStream = response.stream
    .transform(const Utf8Decoder())
    .transform(const JsonEventDecoder())
    .flatten()
    .transform(JsonObjectArrayDecoder())
    .map((m) => Product.fromJson(map));

await for (final product in productsStream) {
    doFancyStuffWithProduct(product);
}

It's designed to only deal with JSON arrays, but that's what I needed.