Hello,
I have configured the SQLFileCommitter.xml and the content is
<committer class = "com.norconex.committer.sql.SQLCommitter"> <fieldMappings> <mapping fromField = "document.reference" toField = "document_reference"/> <mapping fromField="content" toField="content"/> </fieldMappings> </committer>
And I have set the SQLCommitterConfig by using the methods:
`SQLCommitterConfig sqlCommitterConfig = new SQLCommitterConfig();
sqlCommitterConfig.setDriverClass("com.mysql.cj.jdbc.Driver");
sqlCommitterConfig.setConnectionUrl("jdbc:mysql://localhost:3306/sys");
sqlCommitterConfig.setCredentials(new Credentials().setUsername("root").setPassword("123456"));
sqlCommitterConfig.setTableName("test");
sqlCommitterConfig.setPrimaryKey("id");
sqlCommitterConfig.setFixFieldNames(true);
sqlCommitterConfig.setFixFieldValues(true);
sqlCommitterConfig.setTargetContentField("content");
XML xml = new XML(Path.of("fraud\\src\\main\\resources\\SQLFileCommitter.xml"));
SQLCommitter sqlCommitter = new SQLCommitter(sqlCommitterConfig);
sqlCommitter.loadFromXML(xml);
sqlCommitter.loadCommitterFromXML(xml);
crawlerConfig.setCommitters(sqlCommitter);`
I want to store only part of the extracted fields into MYSQL. But I got the exception
Caused by: java.sql.SQLException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '-Backend-Response,Server,x-webkit-csp,Content-Location,x-frame-options,x-cdn-pro' at line 1 Query: INSERT INTO test(X-Backend-Response,Server,x-webkit-csp,Content-Location,x-frame-options,x-cdn-provider,Referrer-Policy,X-SecNG-Response,dc:title,Content-Encoding,Set-Cookie,collector.depth,id,surrogate-control,document_reference,google-site-verification,document.contentEncoding,strict-transport-security,pragma,x-xss-protection,x-idc-id,Cache-Control,document.contentType,Content-Language,expires,document.contentFamily,renderer,collector.redirect-trail,force-rendering,description,title,content,x-edge-timing,x-content-security-policy,X-Cache-Lookup,collector.is-crawl-new,collector.http-fetcher,Content-Length,Content-Type,Transfer-Encoding,X-Parsed-By,Connection,Date,X-UA-Compatible,content-security-policy,x-content-type-options,viewport,x-lb-timing,X-NWS-LOG-UUID,Vary) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) Parameters: [0.028, CLOUD ELB 1.0.0, default-src * blob:; img-src * data: blob: resource: t.captcha.qq.com *.dun.163yun.com *.dun.163.com *.126.net *.nosdn.127.net nos.netease.com;
It seemed like the insert query was too long and the fileds mapping didn't work, so I didn't have enough columns to store. How can I deal with that?
Hello,
I have configured the SQLFileCommitter.xml and the content is
<committer class = "com.norconex.committer.sql.SQLCommitter"> <fieldMappings> <mapping fromField = "document.reference" toField = "document_reference"/> <mapping fromField="content" toField="content"/> </fieldMappings> </committer>And I have set the SQLCommitterConfig by using the methods:
`SQLCommitterConfig sqlCommitterConfig = new SQLCommitterConfig();
I want to store only part of the extracted fields into MYSQL. But I got the exception
Caused by: java.sql.SQLException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '-Backend-Response,Server,x-webkit-csp,Content-Location,x-frame-options,x-cdn-pro' at line 1 Query: INSERT INTO test(X-Backend-Response,Server,x-webkit-csp,Content-Location,x-frame-options,x-cdn-provider,Referrer-Policy,X-SecNG-Response,dc:title,Content-Encoding,Set-Cookie,collector.depth,id,surrogate-control,document_reference,google-site-verification,document.contentEncoding,strict-transport-security,pragma,x-xss-protection,x-idc-id,Cache-Control,document.contentType,Content-Language,expires,document.contentFamily,renderer,collector.redirect-trail,force-rendering,description,title,content,x-edge-timing,x-content-security-policy,X-Cache-Lookup,collector.is-crawl-new,collector.http-fetcher,Content-Length,Content-Type,Transfer-Encoding,X-Parsed-By,Connection,Date,X-UA-Compatible,content-security-policy,x-content-type-options,viewport,x-lb-timing,X-NWS-LOG-UUID,Vary) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) Parameters: [0.028, CLOUD ELB 1.0.0, default-src * blob:; img-src * data: blob: resource: t.captcha.qq.com *.dun.163yun.com *.dun.163.com *.126.net *.nosdn.127.net nos.netease.com;
It seemed like the insert query was too long and the fileds mapping didn't work, so I didn't have enough columns to store. How can I deal with that?