Kecci Kun: February 2020

This is simple project for golang, elasticsearch, and kibana

Docker

Docker & docker-compose Installation

On Mac: Installation Docker with Mac

Try to check docker version:

$ docker --version
$ docker-compose version

Elasticsearch

Elasticsearch Installation on Docker

Download image of elasticsearch with version 7.6.0:

$ docker pull elasticsearch:7.6.0

Try to run image of elasticsearch to be a container:

$ docker run -d --network:elastic -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name elasticsearch elasticsearch:7.6.0
$ docker ps

Kibana

Kibana Installation on Docker

Download image of kibana with version 7.6.0:

$ docker pull kibana:7.6.0
or
$ docker pull docker.elastic.co/kibana/kibana:7.6.0

Try run image of kibana to be a container:

$ docker run -d --net:elastic -p 5601:5601 -e ELASTICSEARCH_URL=http://localhost:9200 --name kibana kibana:7.6.0
$ docker ps

Docker Compose

Create docker compose for Elasticsearch and Kibana

Create file with name docker-compose.yml and type this example configuration into file:

version: '2.2'
services:
  es01:
    image: elasticsearch:7.6.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02
      - cluster.initial_master_nodes=es01,es02
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic
  es02:
    image: elasticsearch:7.6.0
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01
      - cluster.initial_master_nodes=es01,es02
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
    networks:
      - elastic

  kibana:
    image: kibana:7.6.0
    container_name: kibana
    environment:
      SERVER_NAME: kibana.local
      ELASTICSEARCH_HOSTS: http://es01:9200
    ports: 
       - 5601:5601
    networks:
       - elastic
    depends_on:
       - es01
       - es02

volumes:
  data01:
    driver: local
  data02:
    driver: local

networks:
  elastic:
    driver: bridge

In this case, I'm creating 3 containers:

es01 (elasticsearch for node 1)
es02 (elasticsearch for node 2)
kibana (as dashboard)

Run Docker Compose

To run docker compose:

$ docker-compose run -d
$ docker ps

Make sure all containers running

golang

Golang Installation

Install golang Install golang

Try to check version

$ go version

Setup Project

This is golang library to integrate with elasticsearch

First, download library for go-elasticsearch, in here we are using version v7:

$ go mod init
$ go get github.com/elastic/go-elasticsearch/v7

Then create file main.go and insert this example code:

package main

import (
 "crypto/tls"
 "log"
 "net"
 "net/http"
 "time"

 "github.com/elastic/go-elasticsearch/v7"
)

func main() {
 cfg := elasticsearch.Config{
  Addresses: []string{
   "http://localhost:9200",
  },
  Username: "foo",
  Password: "bar",
  Transport: &http.Transport{
   MaxIdleConnsPerHost:   10,
   ResponseHeaderTimeout: time.Second,
   DialContext:           (&net.Dialer{Timeout: time.Second}).DialContext,
   TLSClientConfig: &tls.Config{
    MinVersion: tls.VersionTLS11,
   },
  },
 }

 es, _ := elasticsearch.NewClient(cfg)

 res, err := es.Info()

 if err != nil {
  log.Fatalf("Error getting response: %s", err)
 }

 log.Println(res)

 log.Println(es.Indices)
}

Run golang elasticsearch Application

Try to run application:

$ go run main.go

See More in here Documentation of go-elasticsearch

Source:

https://github.com/abyanjksatu/go-elastic-kibana

http://godoc.org/github.com/elastic/go-elasticsearch

This is standard documentation for golang-migrate and mysql driver

About golang-migrate CLI

You can see the migrate CLI documentation usage in here CLI Documentation

Installation golang-migrate CLI

This is example installation for mysql driver migrate. you can see another database in here Golang Migrate Documentation

$ go get -u -d github.com/golang-migrate/migrate/cmd/migrate github.com/go-sql-driver/mysql
$ go build -tags 'mysql' -o /usr/local/bin/migrate github.com/golang-migrate/migrate/cmd/migrate

Migrate CLI usage

You can see the documentation usage with command of migrate -help or you can see in here Migrate CLI Usage

Run Migration

$ migrate -database mysql://user:password@/dbname -path ./migrations up N

Rollback Migration

$ migrate -database mysql://user:passwrod@/dbname -path ./migrations down N

Recover Failed Migration

$ migrate -database mysql://user:password@/dbname -path ./migrations force N

Note: N is version that you wanted, if you want to execute all you can remove N

Migrations Best Practices

This best practices based on reference in here Migration Version Format

Migration Filename Format

migrate expects the filenames of migrations to have the format:

{version}_{title}.up.{extension}
{version}_{title}.down.{extension}

Common versioning schemes include incrementing integers:

1_user.down.sql
1_user.up.sql
2_role.down.sql
2_role.up.sql
...

Or timestamps at an appropriate resolution:

1500360784_user.down.sql
1500360784_user.up.sql
1500445949_role.down.sql
1500445949_role.up.sql
...

SQL Statement for Migration Files

For example up file 1_user.up.sql

CREATE TABLE IF NOT EXISTS user (
    id INTEGER NOT NULL AUTO_INCREMENT KEY,
    role_id INTEGER,
    user_image_path VARCHAR (250),
    first_name VARCHAR (100),
    second_name VARCHAR (100),
    email VARCHAR (30) UNIQUE,
    password VARCHAR (30),
    phone_number VARCHAR (20),
    address VARCHAR (250),
    created_by VARCHAR(100),
    created_at DATETIME,
    updated_by VARCHAR(100),
    updated_at DATETIME
);

For example down file 1_user.down.sql

DROP TABLE IF EXISTS user;

See more documentation SQL Statement in here SQL Statement