Support for Elastic bulk import

[wingZero21]

Hi,

Do you know if this feature is going to be added soon? We are currently looking at using this tool to extract logs from Cloudflare and it would be fantastic if it offered this option.

Cheers

[jacobbednarz]

I'm not sure about direct Elastic import but Cloudflare do offer Logpush for Enterprise customers (see https://support.cloudflare.com/hc/en-us/articles/360013599871-Getting-Started-with-Cloudflare-Logpush-Beta). If you have an Enterprise account with Cloudflare, you can have them sync the ELS entries to a S3 or Google Storage bucket of your choosing and then ingest them from there.

[wingZero21]

Hi,

Its been mentioned about this option which we might look into, however once the logs have been put into the bucket, how are they then ingested via elastic? Are they being put into a Elastic friendly format?

[jacobbednarz]

To ingest the logs, you'll need to use the appropriate S3 or Google Cloud bucket plugin configured for your Elastic installation. The logs are gzipped, JSON format which is pretty well handled by Elastic installations.

[sbarrier]

@wingZero21An alternative way is to use esbulk to import in ES see https://github.com/miku/esbulk