Elasticsearch github

Elasticsearch github DEFAULT

Introduction

This is a collection of examples to help you get familiar with the Elastic Stack. Each example folder includes a README with detailed instructions for getting up and running with the particular example. The following information pertains to the examples repo as a whole.

Contents

Quick start

You have a few options to get started with the examples:

  • If you want to try them all, you can download the entire repo . Or, if you are familiar with Git, you can clone the repo. Then, simply follow the instructions in the individual README of the examples you're interested in to get started.

  • If you are only interested in a specific example or two, you can download the contents of just those examples - follow instructions in the individual READMEs OR you can use some of the options mentioned here.

Contributing

See here

Example catalog

Below is the list of examples available in this repo:

Common Data Formats

Exploring Public Datasets

Examples using the Elastic Stack for analyzing public dataset.

Getting Started with Graph exploration

Alerting on Elastic Stack

Alerting lets you set up watches (or rules) to detect and alert on changes in your Elasticsearch data. Below is a list of examples watches that configured to detect and alert on a few common scenarios:

Machine learning

Search & API Examples

Security Analytics

Miscellaneous

Sours: https://github.com/elastic/examples

GitHub: Accelerating software development

Enterprise Search
Software and Technology

AT A GLANCE

  • 2

    billion documents

  • 8

    million code repositories

  • 4

    million active users

The Challenge

How do you satisfy the search needs of GitHub's 4 million users while simultaneously providing tactical operational insights that help you iteratively improve customer service?

The Solution

By using Elasticsearch to index over 8 million code repositories as well as indexing critical event data

Case Study Highlights

Enable Powerful Search For Both End-Users And Developers

  • Scale out to meet the needs of burgeoning user base by migrating away from Apache Solr to Elasticsearch
  • Index and query almost any type of publicly exposed data
  • Enable deep programmatic search for developer applications
  • Provide near real-time indexing as soon as users upload new data

Leverage Analytics On Search Data

  • Reveal rogue users by querying indexed logging data
  • Find software bugs within the GitHub platform by indexing all alerts, events, logs and tracking the rate of specific code exceptions
  • Make queries that go beyond standard SQL

Sophisticated Searching For Sophisticated Users

Elasticsearch powers search on GitHub, the largest hosted revision control system in the world with a demanding customer base of over 4 million technical users. GitHub uses Elasticsearch to continually index the data from an ever-growing store of over 8 million code repositories, comprising over 2 billion documents. Using Elasticsearch, GitHub was able to let users easily search this data.

"Search is at the core of GitHub," says Tim Pease, an Operations Engineer at GitHub. "If you go to GitHub.com/search you can search through repositories, users, issues, pull requests, and source code."

One goal of GitHub's Elasticsearch implementation is to index everything that is publicly available on GitHub.com and make it easy to find. Of course, full-text searching is fully supported, but searching based on a wide variety of criteria is also possible and dead simple.

"You can search for a project that uses Clojure as the primary language, and has had activity over the past month, and all this functionality is powered by Elasticsearch," says Pease.

Elasticsearch's flexible storage and retrieval formats, which permit both highly structured and loosely structured data to co-exist in search storage, along with Elasticsearch's extensive set of search primitives, made search implementation straightforward. "You can do lots of queries on that data using Elasticsearch that a standard SQL database won't support," notes Pease.

Powering Analytic Insights Behind The Firewall

GitHub utilizes Elasticsearch's combination of search indexing and analytics capability to drive multiple projects. For example, GitHub found that the analysis capabilities of Elasticsearch queries could be used on stored audit and logging data in order to track users' security-related activity.

"Using Elasticsearch queries, we can quickly see every action the user has done," says Pease. "This is a great way to see whether an account has been stolen, hijacked, or whether the user has done something naughty."

When GitHub was looking to track and analyze code exceptions generated by the various software components that power GitHub.com, they originally used a popular NoSQL database. Code exceptions were stored in secondary indexes, and its analysis features were used to analyze exceptions over time with the results stored back into the database.

"It didn't work very well for our use case," remembered Grant Rodgers, a technical staff member at GitHub. "Once we moved everything to Elasticsearch and used its histogram facet queries, everything worked really well."

GitHub uses Elasticsearch's histogram facet query capability, as well as other statistical facets, to track increases in the rate of specific types of code exceptions. That process reveal bugs in their software systems.

"Elasticsearch's histogram facet query capability performs extremely well. We're looking to expand its use in that particular application," says Rodgers.

Using Elasticsearch queries, we can quickly see every action the user has done - this is a great way to see whether an account has been stolen, hijacked, or whether the user has done something naughty.

– Tim Pease, Operations Engineer, GitHub

Scaling To Millions Of Users

GitHub originally used Solr for search, but found that Solr couldn't scale effectively and was more difficult to manage.

"As more people started using GitHub, we quickly exceeded the storage space that one Solr cluster and Solr instance could handle," says Pease.

Faced with the choice of sharding its own data in Solr in order to handle the load, or moving to Elasticsearch, the choice was easy.

"We decided to move to Elasticsearch because we figured they could shard things much better than we could," says Pease.

Elasticsearch offers automatic shard rebalancing to increase performance and handle failover conditions. Replica shards are automatically distributed to new nodes in a cluster and, in the case of node failure, shards are automatically migrated from failed nodes to good nodes.

Advanced Sharding for High Performance

With over 2 billion documents, all indexed by Elasticsearch, and with users constantly uploading and modifying code, search performance is a key metric for the GitHub team. GitHub serves, on average, 300 search requests per minute.

GitHub uses Elasticsearch to index new code as soon as users push it to a repository on GitHub. The data can be searched on very soon after, and search results are returned for both public repositories, and, for logged-in users, any private repositories they can access.

To optimize access to search data, GitHub uses sharding extensively.

In GitHub's main Elasticsearch cluster, they have about 128 shards, with each shard storing about 120 gigabytes each.

To optimize search within a single repository, GitHub uses the Elasticsearch routing parameter based on the repository ID.

"That allows us to put all the source code for a single repository on one shard," says Pease. "If you're on just a single repository page, and you do a search there, that search actually hits just one shard. Those queries are about twice as fast as searches from the main GitHub search page."

If you're on just a single repository page, and you do a search there, that search actually hits just one shard. Those queries are about twice as fast as searches from the main GitHub search page.

– Tim Pease, Operations Engineer, GitHub

GitHub’s benefits using Elasticsearch

Scale Effectively

GitHub uses Elasticsearch's robust sharding and advanced queries to serve up search across data in 4 million users' code repositories

Analytics via Advanced Queries

GitHub uses Elasticsearch's histogram facet queries, as well as other Elasticsearch analytic queries, to monitor their internal infrastructure for abuse, bugs and more.

High Performance

GitHub uses Elasticsearch's routing parameter and flexible sharding schemes to perform searches within a single repository on a single shard, doubling the speed at which results are served.

Satisfies Users and Developers

Elasticsearch satisfies the search needs of both regular users, and, via the Elasticsearch API, application developers as well.

Sours: https://www.elastic.co/customers/github
  1. Spokane craigslist general jobs
  2. Aquapolis set list
  3. Walmart elk
  4. Fha 251

go-elasticsearch

The official Go client for Elasticsearch.

GoDocGo Report Cardcodecov.ioBuildUnitIntegrationAPI

Compatibility

Language clients are forward compatible; meaning that clients support communicating with greater minor versions of Elasticsearch.

Elastic language clients are also backwards compatible with lesser supported minor Elasticsearch versions.

When using Go modules, include the version in the import path, and specify either an explicit version or a branch:

It's possible to use multiple versions of the client in a single project:

The branch of the client is compatible with the current branch of Elasticsearch.

Installation

Add the package to your file:

Or, clone the repository:

A complete example:

Usage

The package ties together two separate packages for calling the Elasticsearch APIs and transferring data over HTTP: and , respectively.

Use the function to create the client with the default settings.

NOTE: It is critical to both close the response body and to consume it, in order to re-use persistent TCP connections in the default HTTP transport. If you're not interested in the response body, call .

When you export the environment variable, it will be used to set the cluster endpoint(s). Separate multiple adresses by a comma.

To set the cluster endpoint(s) programatically, pass a configuration object to the function.

To set the username and password, include them in the endpoint URL, or use the corresponding configuration options.

To set a custom certificate authority used to sign the certificates of cluster nodes, use the configuration option.

To configure other HTTP settings, pass an object in the configuration object.

See the and files for more examples of configuration and customization of the client. See the for an example of a security configuration.

The following example demonstrates a more complex usage. It fetches the Elasticsearch version from the cluster, indexes a couple of documents concurrently, and prints the search results, using a lightweight wrapper around the response body.

As you see in the example above, the package allows to call the Elasticsearch APIs in two distinct ways: either by creating a struct, such as , and calling its method by passing it a context and the client, or by calling the function on the client directly, using the option functions such as . See more information and examples in the package documentation.

The package handles the transfer of data to and from Elasticsearch, including retrying failed requests, keeping a connection pool, discovering cluster nodes and logging.

Read more about the client internals and usage in the following blog posts:

Helpers

The package provides convenience helpers for working with the client. At the moment, it provides the and the helpers.

Examples

The folder contains a number of recipes and comprehensive examples to get you started with the client, including configuration and customization of the client, using a custom certificate authority (CA) for security (TLS), mocking the transport for unit tests, embedding the client in a custom type, building queries, performing requests individually and in bulk, and parsing the responses.

License

(c) 2019 Elasticsearch. Licensed under the Apache License, Version 2.0.

Sours: https://pkg.go.dev/github.com/elastic/go-elasticsearch/v7
Elasticsearch i Kibana czyli SIEM? - Webinar

Elasticsearch Node.js client

js-standard-styleBuild StatuscodecovNPM downloads

The official Node.js client for Elasticsearch.


Note: In the past months we have worked on the new Elasticsearch Node.js client and you can use it by following the instructions below. If you're going to use the legacy one or report an issue, however, please check out elastic/elasticsearch-js-legacy.


Features

  • One-to-one mapping with REST API.
  • Generalized, pluggable architecture.
  • Configurable, automatic discovery of cluster nodes.
  • Persistent, Keep-Alive connections.
  • Load balancing across all available nodes.
  • Child client support.
  • TypeScript support out of the box.

Install

Node.js support

NOTE: The minimum supported version of Node.js is .

The client versioning follows the Elastic Stack versioning, this means that major, minor, and patch releases are done following a precise schedule that often does not coincide with the Node.js release times.

To avoid support insecure and unsupported versions of Node.js, the client will drop the support of EOL versions of Node.js between minor releases. Typically, as soon as a Node.js version goes into EOL, the client will continue to support that version for at least another minor release. If you are using the client with a version of Node.js that will be unsupported soon, you will see a warning in your logs (the client will start logging the warning with two minors in advance).

Unless you are always using a supported version of Node.js, we recommend defining the client dependency in your with the instead of . In this way, you will lock the dependency on the minor release and not the major. (for example, instead of ).

Node.js VersionNode.js EOL dateEnd of support
(early 2021)
(mid 2021)

Compatibility

Language clients are forward compatible; meaning that clients support communicating with greater or equal minor versions of Elasticsearch. Elasticsearch language clients are only backwards compatible with default distributions and without guarantees made.

Elasticsearch VersionClient Version

To install a specific major of the client, run the following command:

Browser

WARNING: There is no official support for the browser environment. It exposes your Elasticsearch instance to everyone, which could lead to security issues. We recommend that you write a lightweight proxy that uses this client instead, you can see a proxy example here.

Documentation

Quick start

First of all, require the client and initialize it:

const{ Client }=require('@elastic/elasticsearch')constclient=newClient({node: 'http://localhost:9200'})

You can use both the callback-style API and the promise-style API, both behave the same way.

// promise APIconstresult=awaitclient.search({index: 'my-index',body: {query: {match: {hello: 'world'}}}})// callback APIclient.search({index: 'my-index',body: {query: {match: {hello: 'world'}}}},(err,result)=>{if(err)console.log(err)})

The returned value of every API call is formed as follows:

{ body: object|boolean statusCode: number headers: object warnings: [string] meta: object}

Let's see a complete example!

'use strict'const{ Client }=require('@elastic/elasticsearch')constclient=newClient({node: 'http://localhost:9200'})asyncfunctionrun(){// Let's start by indexing some dataawaitclient.index({index: 'game-of-thrones',// type: '_doc', // uncomment this line if you are using Elasticsearch ≤ 6body: {character: 'Ned Stark',quote: 'Winter is coming.'}})awaitclient.index({index: 'game-of-thrones',// type: '_doc', // uncomment this line if you are using Elasticsearch ≤ 6body: {character: 'Daenerys Targaryen',quote: 'I am the blood of the dragon.'}})awaitclient.index({index: 'game-of-thrones',// type: '_doc', // uncomment this line if you are using Elasticsearch ≤ 6body: {character: 'Tyrion Lannister',quote: 'A mind needs books like a sword needs a whetstone.'}})// here we are forcing an index refresh, otherwise we will not// get any result in the consequent searchawaitclient.indices.refresh({index: 'game-of-thrones'})// Let's search!const{ body }=awaitclient.search({index: 'game-of-thrones',// type: '_doc', // uncomment this line if you are using Elasticsearch ≤ 6body: {query: {match: {quote: 'winter'}}}})console.log(body.hits.hits)}run().catch(console.log)

Install multiple versions

If you are using multiple versions of Elasticsearch, you need to use multiple versions of the client. In the past, install multiple versions of the same package was not possible, but with , you can do that via aliasing.

The command you must run to install different version of the client is:

npm install <alias>@npm:@elastic/[email protected]<version>

So for example if you need to install and , you will run

And your will look like the following:

"dependencies": { "es6": "npm:@elastic/[email protected]^6.7.0", "es7": "npm:@elastic/[email protected]^7.0.0" }

You will require the packages from your code by using the alias you have defined.

const{Client: Client6}=require('es6')const{Client: Client7}=require('es7')constclient6=newClient6({node: 'http://localhost:9200'})constclient7=newClient7({node: 'http://localhost:9201'})client6.info(console.log)client7.info(console.log)

Finally, if you want to install the client for the next version of Elasticsearch (the one that lives in Elasticsearch’s master branch), you can use the following command:

npm install [email protected]:elastic/elasticsearch-js

License

This software is licensed under the Apache 2 license.

Sours: https://github.com/elastic/elasticsearch-js

Github elasticsearch

elasticsearch-php

Build statusLatest Stable VersionTotal Downloads

Official low-level client for Elasticsearch. Its goal is to provide common ground for all Elasticsearch-related code in PHP; because of this it tries to be opinion-free and very extendable.

To maintain consistency across all the low-level clients (Ruby, Python, etc.), clients accept simple associative arrays as parameters. All parameters, from the URI to the document body, are defined in the associative array.

Starting from version , all the endpoints (and namespaces) are autogenerated using the util/GenerateEndpoints.php script. This script reads the Elasticsearch API specs and generates the PHP classes for all the endpoints.

Starting from version we included also the XPack endpoints of Elasticsearch. These APIs are related to:

Features

  • One-to-one mapping with REST API and other language clients
  • Configurable, automatic discovery of cluster nodes
  • Persistent, Keep-Alive connections (within the lifetime of the script)
  • Load balancing (with pluggable selection strategy) across all available nodes. Defaults to round-robin
  • Pluggable connection pools to offer different connection strategies
  • Generalized, pluggable architecture - most components can be replaced with your own custom class if specialized behavior is required
  • Option to use asynchronous future, which enables parallel execution of curl requests to multiple nodes

Note:X-Pack endpoints are included from elasticsearch-php 7.7+.

Version Matrix

Elasticsearch VersionElasticsearch-PHP Branch
>= 7.x7.x
>= 6.6, < 7.06.7.x
>= 6.0, < 6.66.5.x
>= 5.0, < 6.05.0
>= 2.0, < 5.01.0 or 2.0
>= 1.0, < 2.01.0 or 2.0
<= 0.90.x0.4
  • If you are using Elasticsearch 7.x you can use Elasticsearch-PHP 7.x branch
  • If you are using Elasticsearch 6.6 to 6.7, use Elasticsearch-PHP 6.7.x branch.
  • If you are using Elasticsearch 6.0 to 6.5, use Elasticsearch-PHP 6.5.x branch.
  • If you are using Elasticsearch 5.x, use Elasticsearch-PHP 5.0 branch.
  • If you are using Elasticsearch 1.x or 2.x, prefer using the Elasticsearch-PHP 2.0 branch. The 1.0 branch is compatible however.
  • If you are using a version older than 1.0, you must install the Elasticsearch-PHP branch. Since ES 0.90.x and below is now EOL, the corresponding branch will not receive any more development or bugfixes. Please upgrade.
  • You should never use Elasticsearch-PHP Master branch, as it tracks Elasticsearch master and may contain incomplete features or breaks in backwards compatibility. Only use ES-PHP master if you are developing against ES master for some reason.

Compatibility

Language clients are forward compatible; meaning that clients support communicating with greater or equal minor versions of Elasticsearch. Elasticsearch language clients are only backwards compatible with default distributions and without guarantees made.

Documentation

Full documentation can be found here. Docs are stored within the repo under /docs/, so if you see a typo or problem, please submit a PR to fix it!

We also provide a code examples generator for PHP using the script. This command parses the file produced from this JSON specification and it generates the PHP examples for each digest value. The examples are stored in asciidoc format under folder.

Installation via Composer

The recommended method to install Elasticsearch-PHP is through Composer.

  1. Add as a dependency in your project's file (change version to suit your version of Elasticsearch, for instance for ES 7.0):

    { "require": { "elasticsearch/elasticsearch": "^7.0" } }
  2. Download and install Composer:

    curl -s http://getcomposer.org/installer | php
  3. Install your dependencies:

    php composer.phar install
  4. Require Composer's autoloader

    Composer also prepares an autoload file that's capable of autoloading all the classes in any of the libraries that it downloads. To use it, just add the following line to your code's bootstrap process:

    <?phpuseElasticsearch\ClientBuilder; require'vendor/autoload.php'; $client = ClientBuilder::create()->build();

You can find out more on how to install Composer, configure autoloading, and other best-practices for defining dependencies at getcomposer.org.

PHP Version Requirement

Version 7.0 of this library requires at least PHP version 7.1. In addition, it requires the native JSON extension to be version 1.3.7 or higher.

Elasticsearch-PHP BranchPHP Version
7.0>= 7.1.0
6.0>= 7.0.0
5.0>= 5.6.6
2.0>= 5.4.0
0.4, 1.0>= 5.3.9

Quickstart

Index a document

In elasticsearch-php, almost everything is configured by associative arrays. The REST endpoint, document and optional parameters - everything is an associative array.

To index a document, we need to specify three pieces of information: index, id and a document body. This is done by constructing an associative array of key:value pairs. The request body is itself an associative array with key:value pairs corresponding to the data in your document:

$params = [ 'index' => 'my_index', 'id' => 'my_id', 'body' => ['testField' => 'abc'] ]; $response = $client->index($params); print_r($response);

The response that you get back indicates the document was created in the index that you specified. The response is an associative array containing a decoded version of the JSON that Elasticsearch returns:

Array ( [_index] => my_index [_type] => _doc [_id] => my_id [_version] => 1 [result] => created [_shards] => Array ( [total] => 1 [successful] => 1 [failed] => 0 ) [_seq_no] => 0 [_primary_term] => 1 )

Get a document

Let's get the document that we just indexed. This will simply return the document:

$params = [ 'index' => 'my_index', 'id' => 'my_id' ]; $response = $client->get($params); print_r($response);

The response contains some metadata (index, version, etc.) as well as a field, which is the original document that you sent to Elasticsearch.

Array ( [_index] => my_index [_type] => _doc [_id] => my_id [_version] => 1 [_seq_no] => 0 [_primary_term] => 1 [found] => 1 [_source] => Array ( [testField] => abc ) )

If you want to retrieve the field directly, there is the method:

$params = [ 'index' => 'my_index', 'id' => 'my_id' ]; $source = $client->getSource($params); print_r($source);

The response will be just the value:

Array ( [testField] => abc )

Search for a document

Searching is a hallmark of Elasticsearch, so let's perform a search. We are going to use the Match query as a demonstration:

$params = [ 'index' => 'my_index', 'body' => [ 'query' => [ 'match' => [ 'testField' => 'abc' ] ] ] ]; $response = $client->search($params); print_r($response);

The response is a little different from the previous responses. We see some metadata (, , etc.) and an array named . This represents your search results. Inside of is another array named , which contains individual search results:

Array ( [took] => 33 [timed_out] => [_shards] => Array ( [total] => 1 [successful] => 1 [skipped] => 0 [failed] => 0 ) [hits] => Array ( [total] => Array ( [value] => 1 [relation] => eq ) [max_score] => 0.2876821 [hits] => Array ( [0] => Array ( [_index] => my_index [_type] => _doc [_id] => my_id [_score] => 0.2876821 [_source] => Array ( [testField] => abc ) ) ) ) )

Delete a document

Alright, let's go ahead and delete the document that we added previously:

$params = [ 'index' => 'my_index', 'id' => 'my_id' ]; $response = $client->delete($params); print_r($response);

You'll notice this is identical syntax to the syntax. The only difference is the operation: instead of . The response will confirm the document was deleted:

Array ( [_index] => my_index [_type] => _doc [_id] => my_id [_version] => 2 [result] => deleted [_shards] => Array ( [total] => 1 [successful] => 1 [failed] => 0 ) [_seq_no] => 1 [_primary_term] => 1 )

Delete an index

Due to the dynamic nature of Elasticsearch, the first document we added automatically built an index with some default settings. Let's delete that index because we want to specify our own settings later:

$deleteParams = [ 'index' => 'my_index' ]; $response = $client->indices()->delete($deleteParams); print_r($response);

The response:

Array ( [acknowledged] => 1 )

Create an index

Now that we are starting fresh (no data or index), let's add a new index with some custom settings:

$params = [ 'index' => 'my_index', 'body' => [ 'settings' => [ 'number_of_shards' => 2, 'number_of_replicas' => 0 ] ] ]; $response = $client->indices()->create($params); print_r($response);

Elasticsearch will now create that index with your chosen settings, and return an acknowledgement:

Array ( [acknowledged] => 1 )
useGuzzleHttp\Ring\Client\MockHandler; useElasticsearch\ClientBuilder; // The connection class requires 'body' to be a file stream handle// Depending on what kind of request you do, you may need to set more values here$handler = newMockHandler([ 'status' => 200, 'transfer_stats' => [ 'total_time' => 100 ], 'body' => fopen('somefile.json'), 'effective_url' => 'localhost' ]); $builder = ClientBuilder::create(); $builder->setHosts(['somehost']); $builder->setHandler($handler); $client = $builder->build(); // Do a request and you'll get back the 'body' response above

If you want to contribute to this project you need to subscribe to a Contributor Agreement. If you want to send a PR for version please use the branch. For instance if you want to send a PR for elasticsearch-php 7 use the branch.

Never send PR to unless you want to contribute to the development version of the client ( represents the next major version).

Each PR should include a unit test using PHPUnit. If you are not familiar with PHPUnit you can have a look at this reference.

That was just a crash-course overview of the client and its syntax. If you are familiar with Elasticsearch, you'll notice that the methods are named just like REST endpoints.

You'll also notice that the client is configured in a manner that facilitates easy discovery via the IDE. All core actions are available under the object (indexing, searching, getting, etc.). Index and cluster management are located under the and objects, respectively.

Check out the rest of the Documentation to see how the entire client works.

Available Licenses

Starting with version 1.3.1, Elasticsearch-PHP is available under two licenses: Apache v2.0 and LGPL v2.1. Versions prior to 1.3.1 are still licensed with only Apache v2.0.

The user may choose which license they wish to use. Since there is no discriminating executable or distribution bundle to differentiate licensing, the user should document their license choice externally, in case the library is re-distributed. If no explicit choice is made, assumption is that redistribution obeys rules of both licenses.

Contributions

All contributions to the library are to be so that they can be licensed under both licenses.

Apache v2.0 License:

Copyright 2013-2016 Elasticsearch

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

LGPL v2.1 Notice:

Copyright (C) 2013-2016 Elasticsearch

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA

Sours: https://github.com/elastic/elasticsearch-php
Elasticsearch i Kibana czyli SIEM? - Webinar

Open Distro for Elasticsearch

Latest

Forward to OpenSearch

For future news regarding this project and OpenSearch, please visit OpenSearch.org. In the coming weeks this website (https://opendistro.github.io/for-elasticsearch/) will be selectively decommissioned and/or forwarded to the corresponding OpenSearch.org content.

Introducing OpenSearch

From the AWS Open Source Blog: Today, we are introducing the OpenSearch project, a community-driven, open source fork of Elasticsearch and Kibana. We are making a long-term investment in OpenSearch to ensure users continue to...

Open Distro for Elasticsearch 1.13.2 is now available

We are delighted to release Open Distro for Elasticsearch 1.13.2 including general availability of Trace Analytics functionality. This release also resolves a bug related to several issues in index management. Open Distro for Elasticsearch 1.13.2...

Getting involved with ODFE-CLI

Open Distro for Elasticsearch Command Line Interface (ODFE-CLI) is a tool that lets you manage your Open Distro for Elasticsearch cluster from the command line and automate tasks. This CLI was designed to have one...

Read more posts on our blog.

Sours: https://opendistro.github.io/for-elasticsearch/

You will also be interested:

Connector configuration instructions provided in this guide apply both to GitHub and GitHub Enterprise Server.

GitHub is a development platform, version control and collaboration platform for teams of all sizes. From open source to business, you can host and review code, manage projects, and build software across departments and continents. The GitHub connectors provided with Workplace Search automatically capture, sync and index the following items:

Issues

Including ID, Content, Status, Repository, Created By, and timestamps

Pull Requests

Including ID, Content, Status, Repository, Created By, and timestamps

Repositories

Including ID, Name, and README content as description. Repository content (e.g. code, files, wiki) is excluded.

Workplace Search supports GitHub Enterprise Server version: 2.19.4

Configuring the GitHub Connectoredit

Configuring the GitHub connector is the first step prior to connecting the GitHub service to Workplace Search, and requires that you create an OAuth App from the GitHub platform. To get started, first log in to GitHub and access your administrative dashboard:


Step 1. Locate the Account drop-down menu in the top-right area and navigate to Settings:

Figure 22. Connecting GitHub


Step 2. From here, you’ll see a set of menu items on the left under Personal Settings. Click Developer Settings:

Figure 23. Connecting GitHub


Workplace Search needs an OAuth App with which to interface. There are two important things to understand before you create one:

  1. The app can stay in developer mode. You do not need to publish it.
  2. Make sure that you create this app with a trusted and stable GitHub account.

We recommend creating a team-owned account for only this app. If access is lost, a new one must be created and the configuration updated in Workplace Search.


Step 3. Click New OAuth App:

Figure 24. Connecting GitHub

You’ll need to fill in specific values:

  • Application Name: A name to help you identify the application. It’s best to make it explicit: Workplace Search, or something like that.
  • Description (Optional): More information will help you remember the application’s purpose.
  • Homepage URL: The base URL of the user interface used to manage Workplace Search (scheme + host). This is affected by which user interface you are using to manage Enterprise Search. Enterprise Search in Kibana and standalone Enterprise Search use different base URLs. See user interfaces for details on each UI.

    For Enterprise Search in Kibana, this should correspond to the value of in your .

    For standalone Enterprise Search this will be the base URL of Enterprise Search.

    Examples:

    # Deployment using a custom domain name https://www.example.com # Deployment using a default Elastic Cloud domain name and Standalone Enterprise Search https://c3397e558e404195a982cb68e84fbb42.ent-search.us-east-1.aws.found.io:443 # Unsecured local development environment for Standalone Enterprise Search http://localhost:3002 # Deployment using a default Elastic Cloud domain name and Enterprise Search in Kibana https://c3397e558e404195a982cb68e84fbb42.kb.us-east-1.aws.found.io:443
  • Authorization callback URL: Among other factors, the Authorization callback URL is affected by which user interface you are using to manage Enterprise Search. Enterprise Search in Kibana and standalone Enterprise Search use different callback URLs. See user interfaces for details on each UI.

    When using Standalone Enterprise Search, use the following URL, substituting with the base URL at which Workplace Search is hosted (scheme + host, no path).

    Examples:

    # Deployment using a custom domain name https://www.example.com/ws/ # Deployment using a default Elastic Cloud domain name https://c3397e558e404195a982cb68e84fbb42.ent-search.us-east-1.aws.found.io:443/ws/ # Unsecured local development environment http://localhost:3002/ws/

    When using Enterprise Search in Kibana, use the following URL, substituting with the base URL of your Kibana instance. This should correspond with the value of in your :

    <KIBANA_BASE_URL>/app/enterprise_search/workplace_search/sources/added

    Examples:

    # Deployment using a custom domain name for Kibana https://www.example.com/app/enterprise_search/workplace_search/sources/added # Deployment using a default Elastic Cloud domain name for Kibana https://c3397e558e404195a982cb68e84fbb42.kb.us-east-1.aws.found.io:443/app/enterprise_search/workplace_search/sources/added # Unsecured local Kibana environment http://localhost:5601/app/enterprise_search/workplace_search/sources/added

Step 4. Once the form is complete, click Register Application:

Figure 25. Connecting GitHub


Step 5. The app is created, and we may now retrieve the Client ID and Client Secret.

Figure 26. Connecting GitHub


Step 6. From the Workplace Search administrative dashboard’s Sources area, locate GitHub and provide both the Client ID and Client Secret.

Voilà! The GitHub connector is now configured, and ready to be used to synchronize content. In order to capture data, you must now connect a GitHub instance with the adequate authentication credentials.

Connecting GitHub to Workplace Searchedit

Once the GitHub connector has been configured, you may connect a GitHub instance to your organization.


Step 1. Head to your organization’s Workplace Search administrative dashboard, and locate the Sources tab.


Step 2. Click Add a new source.


Step 3. Select GitHub (or GitHub Enterprise) in the Configured Sources list, and follow the GitHub authentication flow as presented.


Step 4. Upon the successful authentication flow, you will be redirected to Workplace Search, and will be prompted to select the Organization you would like to synchronize.

GitHub content will now be captured and will be ready for search gradually as it is synced. Once successfully configured and connected, the GitHub synchronization automatically occurs every 2 hours.

Adding GitHub requires that you belong to and have OAuth permissions within a GitHub organization, usually as a GitHub organization admin-level user.

The following table lists the fields synchronized from the connected source to Workplace Search. The attributes in the table apply to the default search application, as follows:

  • Display name - The label used when displayed in the UI
  • Field name - The name of the underlying field attribute
  • Faceted filter - whether the field is a faceted filter by default, or can be enabled (see also: Customizing filters)
  • Automatic query refinement preceding phrases - The default list of phrases that must precede a value of this field in a search query in order to automatically trigger query refinement. If "None," a value from this field may trigger refinement regardless of where it is found in the query string. If , a value from this field must be the first token(s) in the query string. If , automatic query refinement is not available for this field by default. All fields that have a faceted filter ( or ) can also be configured for automatic query refinement; see also Update a content source, Get a content source’s automatic query refinement details and Customizing filters.
Display nameField nameFaceted filterAutomatic query refinement preceding phrases

Id

No

N.A.

URL

No

N.A.

Title

No

N.A.

Type

Default

None

Body

No

N.A.

Created at

No

N.A.

Updated at

No

N.A.

Last updated

No

N.A.

Slug

No

N.A.

Status

Default

[, , , ]

Repository

Default

N.A.

Created by

Default

[, , , ]

Assigned to

Default

[]

Private

Configurable

N.A.

Sours: https://www.elastic.co/guide/en/workplace-search/current/workplace-search-github-connector.html


1422 1423 1424 1425 1426