Streaming and/or bulk vertex insert

[mattyb149]

Hello all! I'm very much enjoying ArcadeDB and am trying to apply it to a real-world use case of mine. Basically we have a high number/throughput of events (vertices) in our system and I'm asking for the best way to get these events into the DB. Every event has the same schema/properties so if there was a way to compile/streamline a Gremlin query or something, perhaps it would be fast enough to handle the throughput. In JDBC there is PreparedStatement that is analogous to what I'm looking for, maybe a parameterized Gremlin query or something like that? Perhaps related to this: is there a way to bulk upload vertices?

I'll have to create edges that link the events together based on their timestamps, but because of the sorting I figure I'd do that offline/later and not part of the event ingest. Happy to hear any and all ideas! Thanks in advance!

[lvca]

Is your data source static, like a file? If yes what format? We have a pretty sick importer for vertices and edges that work in parallel under the hood.

[mattyb149]

In this system (Apache NiFi) it would be streaming into the database at possibly dozens of thousands or more events per second, it's got its own storage system which is why streaming/bulk ingest hasn't worked well for us with other DBs that want a CSV file on disk for example. For JDBC-enabled DBs we use a PreparedStatement and load up the batch

[lvca]

Apache NiFi, cool! It's in Java, so my advice would be to embed the ArcadeDB database in the same JVM where Apache NiFi is running. In this way, you can TCP/IP and marshaling. Then, I'd use the async API that squeezes all the cores of your CPU.

This example was extracted by a test case in our test suite

arcadedb/engine/src/test/java/performance/PerformanceInsertGraphIndexTest.java

Line 192 in 418ac89

private void createVertices() {

private void createVertices() {
    System.out.println("Start inserting " + VERTICES + " vertices...");

    final long startOfTest = System.currentTimeMillis();

    try {
      database.async().setParallelLevel(PARALLEL);
      database.async().setTransactionUseWAL(false);
      database.async().setTransactionSync(WALFile.FLUSH_TYPE.NO);
      database.async().setCommitEvery(5000);
      database.async().onError(new ErrorCallback() {
        @Override
        public void call(final Throwable exception) {
          LogManager.instance().log(this, Level.SEVERE, "ERROR: " + exception, exception);
          System.exit(1);
        }
      });

      int counter = 0;
      for (; counter < VERTICES; ++counter) {
        final MutableVertex vertex = database.newVertex(VERTEX_TYPE_NAME);
        vertex.set("id", counter);

        database.async().createRecord(vertex, null);

        if (counter % 1_000_000 == 0)
          System.out.println("Inserted " + counter + " vertices in " + (System.currentTimeMillis() - startOfTest) + "ms");
      }

      System.out.println("Inserted " + counter + " vertices in " + (System.currentTimeMillis() - startOfTest) + "ms");

    } finally {
      database.async().waitCompletion();
      final long elapsed = System.currentTimeMillis() - startOfTest;
      System.out.println("Insertion finished in " + elapsed + "ms -> " + (VERTICES / (elapsed / 1000F)) + " ops/sec");
    }
  }

This is able to write 3.5M vertices/s on a 2019 Mac and 1.6M indexed vertices/s.