24小时在线 - 古典小说 - 文学小说 - 关于我们
您的当前位置:首页 > Java > 其它 >

Lucene 四.x Spellcheck使用说明

来源:比特技术 编辑:比特技术网 时间:2018-05-29点击:
导读:Lucene 4.x Spellcheck使用说明  Spellcheck是Lucene新版本的功能
Lucene 4.x Spellcheck使用说明

  Spellcheck是Lucene新版本的功能,在介绍spellcheck之前,我们需要弄清楚Spellcheck支持几种数据源。Spellcheck构造函数需要传入Dictionary接口:

  

package org.apache.lucene.search.spell;
/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

import java.io.IOException;

import org.apache.lucene.search.suggest.InputIterator;

/**
 * A simple interface representing a Dictionary. A Dictionary
 * here is a list of entries, where every entry consists of
 * term, weight and payload.
 * 
 */
public interface Dictionary {

  /**
   * Returns an iterator over all the entries
   * @return Iterator
   */
  InputIterator getEntryIterator() throws IOException;
}

  常用的Dictionary主要有以下几种,常用的主要有基于文本型的和基于lucene索引构建的:

  

  下面是我测试用的一段代码,代码包括索引构建和索引查询:

  

package com.tianditu.com.search;

import java.io.File;
import java.io.IOException;

import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.search.spell.LuceneDictionary;
import org.apache.lucene.search.spell.SpellChecker;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;
import org.apache.lucene.store.MMapDirectory;
import org.apache.lucene.util.Version;

public class GlobalSuggest {

	//拼写检查构建的索引
	private  final String SPELL_CHECK_FOLDER = "c:\\spellcheck\\";
	
	//根据已有的索引
	private final String GLOBAL_PINYIN_SUGGEST = "O:\\searchwork_custom\\data_index\\pinyin2008\\";
	
	//构建索引
	public void testIndexPinyin2008() throws IOException{
		
		long start = System.currentTimeMillis();
		//北京吉威时代软件股份有限公司
		//String indexDir ="O:\\searchwork_custom\\data_index\\GlobalIndex\\";
		Directory direct = new MMapDirectory(new File(GLOBAL_PINYIN_SUGGEST));
		
		
		LuceneDictionary ld = new LuceneDictionary(DirectoryReader.open(direct), "name");
		
		ld.getEntryIterator();
		
		Directory spd = FSDirectory.open(new File(SPELL_CHECK_FOLDER));
		
		
		SpellChecker sc = new SpellChecker(spd);
		//sc.in
		
		IndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_30,null);
		
		//往spellcheck目录下写索引--------------
		sc.indexDictionary(ld, iwc, true);
		
		sc.close();
		
		long end = System.currentTimeMillis();
		System.out.println("索引完毕,耗时:"+(end-start)+"ms");
	}
	
	public void testIndex() throws IOException{
		long start = System.currentTimeMillis();
		//北京吉威时代软件股份有限公司
		String indexDir ="O:\\searchwork_custom\\data_index\\GlobalIndex\\";
		Directory direct = new MMapDirectory(new File(indexDir));
		
		
		LuceneDictionary ld = new LuceneDictionary(DirectoryReader.open(direct), "name");
		
		ld.getEntryIterator();
		
		Directory spd = FSDirectory.open(new File(SPELL_CHECK_FOLDER));
		
		
		SpellChecker sc = new SpellChecker(spd);
		//sc.in
		
		IndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_30,null);
		
		sc.indexDictionary(ld, iwc, true);
		
		sc.close();
		
		long end = System.currentTimeMillis();
		System.out.println("索引完毕,耗时:"+(end-start)+"ms");
	}
	
	
	public void testSearch(String wd) throws IOException{
		
		//构建Directory
		Directory spd = FSDirectory.open(new File(SPELL_CHECK_FOLDER));
		
		//实例化 spellcheck组件
		
		SpellChecker sc = new SpellChecker(spd);
		
		//根据输入关键字  获得N条最相近的几率 第三个鄙视精确度 越大越匹配 安装实际需要调整
		String[] suggests = sc.suggestSimilar(wd, 10,0.6f);
		if(suggests!=null){
			for(String word:suggests){
				System.out.println("Dou you mean:"+word);
			}
		}
	}
	/**
	 * @param args
	 * @throws IOException 
	 */
	public static void main(String[] args) throws IOException {

		
		GlobalSuggest spellcheck = new GlobalSuggest();
		//spellcheck.testIndexPinyin2008();
		spellcheck.testSearch("beijing京鸭");
		
		//spellcheck.testSearch("beijng");

		
	}

}

  其中索引构建处代码:

  

	//构建索引
	public void testIndexPinyin2008() throws IOException{
		
		long start = System.currentTimeMillis();
		//北京吉威时代软件股份有限公司
		//String indexDir ="O:\\searchwork_custom\\data_index\\GlobalIndex\\";
		Directory direct = new MMapDirectory(new File(GLOBAL_PINYIN_SUGGEST));
		
		
		LuceneDictionary ld = new LuceneDictionary(DirectoryReader.open(direct), "name");
		
		ld.getEntryIterator();
		
		Directory spd = FSDirectory.open(new File(SPELL_CHECK_FOLDER));
		
		
		SpellChecker sc = new SpellChecker(spd);
		//sc.in
		
		IndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_30,null);
		
		//往spellcheck目录下写索引--------------
		sc.indexDictionary(ld, iwc, true);
		
		sc.close();
		
		long end = System.currentTimeMillis();
		System.out.println("索引完毕,耗时:"+(end-start)+"ms");
	}

  此处代码,就是根据已有的索引来构建Spellcheck所需的索引。

     Spellcheck查询索引代码片段如下:

  

//构建Directory
		Directory spd = FSDirectory.open(new File(SPELL_CHECK_FOLDER));
		
		//实例化 spellcheck组件
		
		SpellChecker sc = new SpellChecker(spd);
		
		//根据输入关键字  获得N条最相近的几率 第三个鄙视精确度 越大越匹配 安装实际需要调整
		String[] suggests = sc.suggestSimilar(wd, 10,0.6f);
		if(suggests!=null){
			for(String word:suggests){
				System.out.println("Dou you mean:"+word);
			}
		}

 相关算法:默认是 LevensteinDistance 。

     

 

 查询样例:

    1、查询汉字,有错别字情况:

    

    2、查询拼音:

    

    3、拼音汉字夹杂:

    

       (备注:发现问题了,拼音和汉字夹杂的情况不行,如果想使用,需要进行某种处理。)

    4、如果处理一长串汉字,中间夹杂错别字:

    

  总结:看来spellcheck能力还是有限,如果需要用还可能改造。

      

  

  

  

    

来源:比特技术在线(www.byte.online),欢迎分享本文,转载请保留出处!
http://www.byte.online/a/Java/qita/2018/223827.html
责任编辑:比特技术网

    打赏

    取消

    感谢您的支持,我会继续努力的!

    扫码支持
    扫码打赏,你说多少就多少

    打开支付宝扫一扫,即可进行扫码打赏哦

    网友评论:

    Copyright © www.byte.online 比特技术在线 版权所有 粤ICP备15038486号-3
    本站所有资料来源于网络,由个人收集整理,仅限于个人学习和研究,如果侵犯您的权益请联系我们删除!
    Top