https://www.809508860.xyz/tags/hive/ Recent content in Hive on Colben Notes Hugo zh-cn Tue, 03 Feb 2026 11:00:00 +0800 https://www.809508860.xyz/post/hdp3.3/ Fri, 28 Jul 2023 14:00:00 +0800 https://www.809508860.xyz/post/hdp3.3/ <h2 id="环境">环境</h2> <table> <thead> <tr> <th>主机名</th> <th>地址</th> <th>数据目录</th> <th>组件</th> </tr> </thead> <tbody> <tr> <td>hdp-nn</td> <td>192.168.8.1/24</td> <td>/data/hdp-nn</td> <td>Namenode Spark</td> </tr> <tr> <td>hdp-snn</td> <td>192.168.8.2/24</td> <td>/data/hdp-snn</td> <td>SecondaryNamenode</td> </tr> <tr> <td>hdp-rm</td> <td>192.168.8.3/24</td> <td>-</td> <td>ResourceManager</td> </tr> <tr> <td>hdp-worker0</td> <td>192.168.8.10/24</td> <td>/data/hdp-dn</td> <td>Datanode NodeManager Spark</td> </tr> <tr> <td>hdp-worker1</td> <td>192.168.8.11/24</td> <td>/data/hdp-dn</td> <td>Datanode NodeManager Spark</td> </tr> <tr> <td>hive-hs</td> <td>192.168.8.20/24</td> <td>-</td> <td>HiveServer2 Tez</td> </tr> <tr> <td>hive-ms</td> <td>192.168.8.21/24</td> <td>-</td> <td>HiveMetastore Tez</td> </tr> </tbody> </table> <h2 id="部署-hadoop-集群">部署 hadoop 集群</h2> <h3 id="服务器初始配置">服务器初始配置</h3> <ul> <li>在<strong>全部主机</strong>上执行如下操作</li> <li>禁用防火墙</li> <li>禁用 selinux</li> <li>配置时间同步</li> <li>配置主机名解析，修改 /etc/hosts，增加如下内容 <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span><span class="lnt">2 </span><span class="lnt">3 </span><span class="lnt">4 </span><span class="lnt">5 </span><span class="lnt">6 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="c1"># hadoop</span> </span></span><span class="line"><span class="cl">192.168.8.1 hdp-nn </span></span><span class="line"><span class="cl">192.168.8.2 hdp-snn </span></span><span class="line"><span class="cl">192.168.8.3 hdp-dn </span></span><span class="line"><span class="cl">192.168.8.10 hdp-worker0 </span></span><span class="line"><span class="cl">192.168.8.11 hdp-worker1 </span></span></code></pre></td></tr></table> </div> </div></li> </ul> <h3 id="ssh-免密登录">ssh 免密登录</h3> <ul> <li> <p>在 <strong>hdp-nn</strong> 配置 ssh 免密登录 hdp-nn、hdp-snn 和 hdp-workerX</p> https://www.809508860.xyz/post/hdp2/ Tue, 23 May 2023 10:00:00 +0800 https://www.809508860.xyz/post/hdp2/ <h2 id="环境">环境</h2> <table> <thead> <tr> <th>主机名</th> <th>地址</th> <th>数据目录</th> <th>组件</th> </tr> </thead> <tbody> <tr> <td>编译服务器</td> <td>-</td> <td>-</td> <td>各种编译工具</td> </tr> <tr> <td>hdp-nn</td> <td>192.168.8.1/24</td> <td>/data/hdp-nn</td> <td>Namenode Spark</td> </tr> <tr> <td>hdp-snn</td> <td>192.168.8.2/24</td> <td>/data/hdp-snn</td> <td>SecondaryNamenode</td> </tr> <tr> <td>hdp-rm</td> <td>192.168.8.3/24</td> <td>-</td> <td>ResourceManager</td> </tr> <tr> <td>hdp-slave0</td> <td>192.168.8.10/24</td> <td>/data/hdp-dn</td> <td>Datanode NodeManager Spark</td> </tr> <tr> <td>hdp-slave1</td> <td>192.168.8.11/24</td> <td>/data/hdp-dn</td> <td>Datanode NodeManager Spark</td> </tr> <tr> <td>hive-hs</td> <td>192.168.8.20/24</td> <td>-</td> <td>HiveServer2 Tez</td> </tr> <tr> <td>hive-ms</td> <td>192.168.8.21/24</td> <td>-</td> <td>HiveMetastore Tez</td> </tr> <tr> <td>hbase-m</td> <td>192.168.8.30/24</td> <td>-</td> <td>HbaseMaster</td> </tr> <tr> <td>hbase-bm</td> <td>192.168.8.31/24</td> <td>-</td> <td>HbaseBackupMaster</td> </tr> <tr> <td>hbase-rs0</td> <td>192.168.8.32/24</td> <td>-</td> <td>HbaseRigionServer</td> </tr> <tr> <td>hbase-rs1</td> <td>192.168.8.33/24</td> <td>-</td> <td>HbaseRigionServer</td> </tr> </tbody> </table> <h2 id="部署-hadoop-集群">部署 hadoop 集群</h2> <h3 id="服务器初始配置">服务器初始配置</h3> <ul> <li>在<strong>全部主机</strong>上执行如下操作</li> <li>禁用防火墙</li> <li>禁用 selinux</li> <li>配置时间同步</li> <li>配置主机名解析，修改 /etc/hosts，增加如下内容 <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span><span class="lnt">2 </span><span class="lnt">3 </span><span class="lnt">4 </span><span class="lnt">5 </span><span class="lnt">6 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl"># hadoop </span></span><span class="line"><span class="cl">192.168.8.1 hdp-nn </span></span><span class="line"><span class="cl">192.168.8.2 hdp-snn </span></span><span class="line"><span class="cl">192.168.8.3 hdp-dn </span></span><span class="line"><span class="cl">192.168.8.10 hdp-slave0 </span></span><span class="line"><span class="cl">192.168.8.11 hdp-slave1 </span></span></code></pre></td></tr></table> </div> </div></li> </ul> <h3 id="ssh-免密登录">ssh 免密登录</h3> <ul> <li> <p>在 <strong>hdp-nn</strong> 配置 ssh 免密登录 hdp-nn、hdp-snn 和 hdp-slaveX</p> https://www.809508860.xyz/post/sqoop/ Wed, 30 Oct 2019 11:35:48 +0800 https://www.809508860.xyz/post/sqoop/ <h2 id="把-mysql-表数据导入-hive-中">把 mysql 表数据导入 hive 中</h2> <ul> <li>命令格式 <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sqoop import <span class="o">{</span>参数列表<span class="o">}</span> </span></span></code></pre></td></tr></table> </div> </div></li> <li>默认在 hive 中创建指定表，如果该表已存在，则把 mysql 表数据追加到该表中，常规参数如下 <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span><span class="lnt">2 </span><span class="lnt">3 </span><span class="lnt">4 </span><span class="lnt">5 </span><span class="lnt">6 </span><span class="lnt">7 </span><span class="lnt">8 </span><span class="lnt">9 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">--connect jdbc:mysql://_mysql_server_:3306/_mysql_dbname_ </span></span><span class="line"><span class="cl">--username _mysql_user_ </span></span><span class="line"><span class="cl">--password _mysql_password_ </span></span><span class="line"><span class="cl">--table _mysql_tablename_ </span></span><span class="line"><span class="cl">--hive-import </span></span><span class="line"><span class="cl">--hive-database _hive_dbname_ </span></span><span class="line"><span class="cl">--hive-table _hive_tablename_ </span></span><span class="line"><span class="cl">--hive-drop-import-delims </span></span><span class="line"><span class="cl">-m <span class="m">1</span> </span></span></code></pre></td></tr></table> </div> </div></li> <li>如果该表已存在，则报错退出，参数如下 <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">--create-hive-table </span></span></code></pre></td></tr></table> </div> </div></li> <li>如果该表已存在，则覆盖重新导入，参数如下 <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">--hive-overwrite </span></span></code></pre></td></tr></table> </div> </div></li> <li>导入增量数据，参数如下 <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span><span class="lnt">2 </span><span class="lnt">3 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">--incremental append </span></span><span class="line"><span class="cl">--check-column _mysql_tablecolumn_ </span></span><span class="line"><span class="cl">--last-value _last_value_ </span></span></code></pre></td></tr></table> </div> </div></li> </ul> <h2 id="把-hive-表数据导入-mysql-中">把 hive 表数据导入 mysql 中</h2> <ul> <li>命令格式 <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">sqoop <span class="nb">export</span> <span class="o">{</span>参数列表<span class="o">}</span> </span></span></code></pre></td></tr></table> </div> </div></li> <li>mysql 中需提前创建好待导入的表，常规参数如下 <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span><span class="lnt">2 </span><span class="lnt">3 </span><span class="lnt">4 </span><span class="lnt">5 </span><span class="lnt">6 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">--connect jdbc:mysql://_mysql_server_:3306/_mysql_dbname_ </span></span><span class="line"><span class="cl">--username _mysql_username_ </span></span><span class="line"><span class="cl">--password _mysql_password_ </span></span><span class="line"><span class="cl">--table _mysql_tablename_ </span></span><span class="line"><span class="cl">--hcatalog-database _hive_dbname_ </span></span><span class="line"><span class="cl">--hcatalog-table _hive_tablename_ </span></span></code></pre></td></tr></table> </div> </div></li> <li>只导出 hive 表中的部分的指定(部分)字段，参数如下 <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">--columns <span class="s2">&#34;_column_name1_,_column_name2_,_column_name3_&#34;</span> </span></span></code></pre></td></tr></table> </div> </div></li> </ul>