ELK修改字段后重建索引

2020年3月14日11:43:22 发表评论 961 views

今日闲来无事希望将ELK的Nginx访问dashboard美化一下，却发现kibana Visualize中的Metrics需要是number字段才可以使用除count外的统计字段，由于之前并未注意此方面的东西，导致部分需要统计的字段是string类型，需要修改字段类型，但是ELK并不能直接修改字段类型，需要重建索引（相关命令在kibana的Dev Tools =>consle中执行），于是写此篇记录一下过程。

一、环境信息

1、ELK版本

6.8.2

2、nignx日志格式

log_format json '{"@timestamp":"$time_iso8601",'
'"@version":"1",'
'"client":"$remote_addr",'
'"realip":"$http_x_forwarded_for",'
'"url":"$request_uri",'
'"status":"$status",'
'"domain":"$host",'
'"host":"$server_addr",'
'"size":"$body_bytes_sent",'
'"responsetime":"$request_time",'
'"referer":"$http_referer",'
'"ua":"$http_user_agent"'
'}'

3、logstash配置文件内容

input {
  file {
    path => ["/var/log/nginx/access.log"]
    codec => json
  }
}

filter {
    geoip {
      source => "[realip]"
      target => "geoip"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float"]
      # 此处添加convert，将string转化成float
      convert => [ "responsetime", "float"]
    }
}

output {
    elasticsearch {
      hosts => ["1.2.3.4"]
      manage_template => true
      index => "logstash-lb01-%{+YYYY.MM.dd}"
  }
}

二、重建索引流程

1、查找源索引的mapping

ELK修改字段后重建索引

2、创建字段修改后的中间索引

PUT logstash-lb01-2020.03.14_1
# 注意此处新的mapping需要将源索引mapping中冗余的内容去掉后方可创建成功
PUT logstash-lb01-2020.03.14_1/doc/_mapping
{
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "@version": {
      "type": "keyword"
    },
    "client": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "domain": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "geoip": {
      "dynamic": "true",
      "properties": {
        "city_name": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "continent_code": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "coordinates": {
          "type": "float"
        },
        "country_code2": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "country_code3": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "country_name": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "ip": {
          "type": "ip"
        },
        "latitude": {
          "type": "half_float"
        },
        "location": {
          "type": "geo_point"
        },
        "longitude": {
          "type": "half_float"
        },
        "region_code": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "region_name": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "timezone": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    },
    "host": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "path": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "realip": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "referer": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "responsetime": {
      "type": "float" #修改此处的string为float
    },
    "size": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "status": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "tags": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "type": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "ua": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "url": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    }
  }
}

3、停止服务，将旧索引的数据导入到中间索引中

POST _reindex
{
  "source": {
    "index": "logstash-lb01-2020.03.14"
  },
  "dest": {
    "index": "logstash-lb01-2020.03.14_1"
  }
}

4、检查索引是否导入成功

GET /logstash-lb01-2020.03.14/doc/_search
GET /logstash-lb01-2020.03.14_1/doc/_search

5、删除旧索引

DELETE logstash-lb01-2020.03.14

6、创建新索引

PUT logstash-lb01-2020.03.14
PUT logstash-lb01-2020.03.14/doc/_mapping
{
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "@version": {
      "type": "keyword"
    },
    "client": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "domain": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "geoip": {
      "dynamic": "true",
      "properties": {
        "city_name": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "continent_code": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "coordinates": {
          "type": "float"
        },
        "country_code2": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "country_code3": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "country_name": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "ip": {
          "type": "ip"
        },
        "latitude": {
          "type": "half_float"
        },
        "location": {
          "type": "geo_point"
        },
        "longitude": {
          "type": "half_float"
        },
        "region_code": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "region_name": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "timezone": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    },
    "host": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "path": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "realip": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "referer": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "responsetime": {
      "type": "float"
    },
    "size": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "status": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "tags": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "type": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "ua": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "url": {
      "type": "text",
      "norms": false,
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    }
  }
}

7、从中间索引将数据导入到新索引中

POST _reindex
{
  "source": {
    "index": "logstash-lb01-2020.03.14_1"
  },
  "dest": {
    "index": "logstash-lb01-2020.03.14"
  }
}

8、检查索引导入情况

GET /logstash-lb01-2020.03.14/doc/_search

9、删除中间索引

DELETE logstash-lb01-2020.03.14_1

三、重建索引后的工作

1、重新刷新索引

ELK修改字段后重建索引

2、制作Visualize

ELK修改字段后重建索引

3、后记

由于索引是按照每日新建的，所以仅修改当日的索引后刷新索引会导致修改的字段出现冲突，需要将所有的索引都重建后方可完成索引重建，但是由于我这边个人使用，数据不多，为了方便直接将之前的索引全部删掉了，同时生产环境中需要不停服务重建索引可以使用alias的方式重建索引，同时如果数据量很大也需要滚动重建索引，此处个人使用要求不高，所以直接停掉服务重建。

四、参考资料

https://www.cnblogs.com/huangxiufen/p/12461191.html

https://blog.csdn.net/qq_36762677/article/details/97244174

https://www.cnblogs.com/bigben0123/p/10059289.html

https://www.cnblogs.com/Creator/p/3722408.html

https://www.cnblogs.com/minseo/p/10949802.html

https://blog.csdn.net/u010603691/article/details/79310495

http://www.eryajf.net/2367.html

https://www.41sh.cn/?id=23

https://blog.csdn.net/u012107143/article/details/93870999

weinxin
我的微信
如果有技术上的问题可以扫一扫我的微信
版权声明
1. 本网站名称：Leon的博客
2. 本站永久网址：https://blog.leonshadow.cn
3. 本网站的文章部分内容可能来源于网络，仅供大家学习与参考，如有侵权，请联系站长QQ632113590进行删除处理。
4. 本站一切资源不代表本站立场，并不代表本站赞同其观点和对其真实性负责。
5. 本站一律禁止以任何方式发布或转载任何违法的相关信息，访客发现请向站长举报
6. 本站资源大多存储在云盘，如发现链接失效，请联系我们我们会第一时间更新。
liyang