[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 19:55:53 dist-test-slave-tzct krb5kdc[8872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 19:55:53 dist-test-slave-tzct krb5kdc[8872](info): set up 2 sockets
May 01 19:55:53 dist-test-slave-tzct krb5kdc[8872](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20240501 19:55:54.853890 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20240501 19:55:55.057816 8865 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.204s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20240501 19:55:55.263425 8865 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.205s user 0.002s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20240501 19:55:55.489573 8865 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.226s user 0.001s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/test-user.keytab.
W20240501 19:55:56.210136 8865 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.720s user 0.001s sys 0.000s
May 01 19:55:56 dist-test-slave-tzct krb5kdc[8872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T19:55:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:55:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
W20240501 19:55:57.462478 8865 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.8.168.126: real 1.232s user 0.002s sys 0.001s
I20240501 19:55:57.463315 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33165
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:33165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:55:57.573161 8885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:57.573395 8885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:57.573444 8885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:57.576390 8885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:55:57.576443 8885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:55:57.576464 8885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:57.576488 8885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:55:57.576504 8885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:55:57.580648 8885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:33165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:57.581548 8885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:57.582397 8885 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:57.589462 8890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.590014 8893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.590554 8891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.591168 8885 server_base.cc:1034] running on GCE node
I20240501 19:55:57.591665 8885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:57.592638 8885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:57.593840 8885 hybrid_clock.cc:648] HybridClock initialized: now 1714593357593830 us; error 27 us; skew 500 ppm
May 01 19:55:57 dist-test-slave-tzct krb5kdc[8872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593357, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:55:57.602075 8885 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 19:55:57.603093 8885 webserver.cc:460] Webserver started at http://127.8.168.126:37859/ using document root <none> and password file <none>
I20240501 19:55:57.603674 8885 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.603735 8885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.603924 8885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.605808 8885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "352aaed4f9f84ceb8d9a6a67cb6d5e63"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-tzct"
server_key: "f119a26021714127061fcc710ab556bd"
server_key_iv: "3e7da7fdf73e88e999f8a55a259f0fac"
server_key_version: "encryptionkey@0"
I20240501 19:55:57.606289 8885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "352aaed4f9f84ceb8d9a6a67cb6d5e63"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-tzct"
server_key: "f119a26021714127061fcc710ab556bd"
server_key_iv: "3e7da7fdf73e88e999f8a55a259f0fac"
server_key_version: "encryptionkey@0"
I20240501 19:55:57.609824 8885 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240501 19:55:57.612069 8900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.613212 8885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:55:57.613344 8885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "352aaed4f9f84ceb8d9a6a67cb6d5e63"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-tzct"
server_key: "f119a26021714127061fcc710ab556bd"
server_key_iv: "3e7da7fdf73e88e999f8a55a259f0fac"
server_key_version: "encryptionkey@0"
I20240501 19:55:57.613435 8885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.629281 8885 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.632258 8885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:57.632440 8885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.639034 8885 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:33165
I20240501 19:55:57.639048 8952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:33165 every 8 connection(s)
I20240501 19:55:57.639998 8885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:55:57.642872 8953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:57.648068 8953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63: Bootstrap starting.
I20240501 19:55:57.649400 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 8885
I20240501 19:55:57.649513 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:55:57.650493 8953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:57.650517 8865 external_mini_cluster.cc:1428] Setting key db33884a0b5b6b0d2c35e65b209f7c97
I20240501 19:55:57.651208 8953 log.cc:824] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:57.653496 8953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63: No bootstrap required, opened a new log
I20240501 19:55:57.656333 8953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352aaed4f9f84ceb8d9a6a67cb6d5e63" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33165 } }
I20240501 19:55:57.656525 8953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:57.656579 8953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352aaed4f9f84ceb8d9a6a67cb6d5e63, State: Initialized, Role: FOLLOWER
I20240501 19:55:57.657065 8953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352aaed4f9f84ceb8d9a6a67cb6d5e63" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33165 } }
I20240501 19:55:57.657202 8953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:55:57.657269 8953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:55:57.657378 8953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:57.658202 8953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352aaed4f9f84ceb8d9a6a67cb6d5e63" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33165 } }
I20240501 19:55:57.658522 8953 leader_election.cc:302] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 352aaed4f9f84ceb8d9a6a67cb6d5e63; no voters:
I20240501 19:55:57.658798 8953 leader_election.cc:288] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:55:57.658896 8958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:57.659085 8958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [term 1 LEADER]: Becoming Leader. State: Replica: 352aaed4f9f84ceb8d9a6a67cb6d5e63, State: Running, Role: LEADER
I20240501 19:55:57.659430 8958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352aaed4f9f84ceb8d9a6a67cb6d5e63" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33165 } }
I20240501 19:55:57.659862 8953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:55:57.660501 8960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 352aaed4f9f84ceb8d9a6a67cb6d5e63. Latest consensus state: current_term: 1 leader_uuid: "352aaed4f9f84ceb8d9a6a67cb6d5e63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352aaed4f9f84ceb8d9a6a67cb6d5e63" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33165 } } }
I20240501 19:55:57.660611 8960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:57.661168 8959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "352aaed4f9f84ceb8d9a6a67cb6d5e63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352aaed4f9f84ceb8d9a6a67cb6d5e63" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33165 } } }
I20240501 19:55:57.661307 8959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63 [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:57.661581 8963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
May 01 19:55:57 dist-test-slave-tzct krb5kdc[8872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:55:57.664610 8963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:55:57.670372 8963 catalog_manager.cc:1347] Generated new cluster ID: 26175181c59f4a099cf5a395dcdfa1a5
I20240501 19:55:57.670447 8963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:55:57.670987 8956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:55:57.652840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.8.168.126:33165)
0501 19:55:57.653213 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0501 19:55:57.653225 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 19:55:57.653849 (+ 624us) server_negotiation.cc:407] Connection header received
0501 19:55:57.655971 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:55:57.655984 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:55:57.656246 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:55:57.656551 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:55:57.658298 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:55:57.659732 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:55:57.661693 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:55:57.661969 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:55:57.665412 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:55:57.665431 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:55:57.665445 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:55:57.665486 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:55:57.667652 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:55:57.668229 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:55:57.668236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:55:57.668241 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:55:57.668332 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:55:57.668662 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:55:57.668666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:55:57.668668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:55:57.669125 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:55:57.669318 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0501 19:55:57.669858 (+ 540us) server_negotiation.cc:299] Negotiation successful
0501 19:55:57.670108 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":117,"threads_started":1}
I20240501 19:55:57.675977 8963 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:55:57.676801 8963 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:55:57.682125 8963 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 352aaed4f9f84ceb8d9a6a67cb6d5e63: Generated new TSK 0
I20240501 19:55:57.682782 8963 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
I20240501 19:55:57.712050 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33165
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:55:57.816278 8979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:57.816515 8979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:57.816579 8979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:57.819648 8979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:55:57.819715 8979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:57.819799 8979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:55:57.823745 8979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33165
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:57.824522 8979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:57.825340 8979 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:57.833032 8987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.833489 8985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.833928 8984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.834548 8979 server_base.cc:1034] running on GCE node
I20240501 19:55:57.834893 8979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:57.835414 8979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:57.836585 8979 hybrid_clock.cc:648] HybridClock initialized: now 1714593357836571 us; error 27 us; skew 500 ppm
May 01 19:55:57 dist-test-slave-tzct krb5kdc[8872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593357, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:55:57.839268 8979 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 19:55:57.840111 8979 webserver.cc:460] Webserver started at http://127.8.168.65:35699/ using document root <none> and password file <none>
I20240501 19:55:57.840615 8979 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.840669 8979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.840867 8979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.842574 8979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "a62095df3a3b453d94a00dca0bda7e72"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-tzct"
server_key: "bb8f4545276b5ba2ebf5379e2df40e69"
server_key_iv: "37a7b2d6e430aea6805624e5db9eba1d"
server_key_version: "encryptionkey@0"
I20240501 19:55:57.843019 8979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "a62095df3a3b453d94a00dca0bda7e72"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-tzct"
server_key: "bb8f4545276b5ba2ebf5379e2df40e69"
server_key_iv: "37a7b2d6e430aea6805624e5db9eba1d"
server_key_version: "encryptionkey@0"
I20240501 19:55:57.846217 8979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:55:57.848239 8994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.849228 8979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:57.849330 8979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "a62095df3a3b453d94a00dca0bda7e72"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-tzct"
server_key: "bb8f4545276b5ba2ebf5379e2df40e69"
server_key_iv: "37a7b2d6e430aea6805624e5db9eba1d"
server_key_version: "encryptionkey@0"
I20240501 19:55:57.849409 8979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.872205 8979 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.875340 8979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:57.875523 8979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.876102 8979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:57.876960 8979 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:57.877010 8979 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.877063 8979 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:57.877091 8979 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.885812 8979 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:38293
I20240501 19:55:57.885833 9107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:38293 every 8 connection(s)
I20240501 19:55:57.886755 8979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 19:55:57.887241 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 8979
I20240501 19:55:57.887342 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 19:55:57.887601 8865 external_mini_cluster.cc:1428] Setting key 91a56f6f0d417188c1df1db407de2443
May 01 19:55:57 dist-test-slave-tzct krb5kdc[8872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593357, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.168.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.66@KRBTEST.COM" created.
I20240501 19:55:57.899658 8956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:55:57.888574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:38111 (local address 127.8.168.126:33165)
0501 19:55:57.888712 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0501 19:55:57.888732 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0501 19:55:57.889533 (+ 801us) server_negotiation.cc:407] Connection header received
0501 19:55:57.890346 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:55:57.890350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:55:57.890408 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:55:57.890496 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:55:57.892597 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:55:57.893142 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:55:57.893977 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:55:57.894100 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:55:57.896509 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:55:57.896528 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:55:57.896530 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:55:57.896554 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:55:57.897898 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:55:57.898325 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:55:57.898328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:55:57.898329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:55:57.898371 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:55:57.898585 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:55:57.898588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:55:57.898589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:55:57.898736 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:55:57.898835 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0501 19:55:57.899364 (+ 529us) server_negotiation.cc:299] Negotiation successful
0501 19:55:57.899493 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 19:55:57.900166 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:55:57.888936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33165 (local address 127.8.168.65:38111)
0501 19:55:57.889384 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0501 19:55:57.889425 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0501 19:55:57.890155 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:55:57.890670 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:55:57.890682 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:55:57.891210 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:55:57.892449 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:55:57.892470 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:55:57.893266 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:55:57.893270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:55:57.893860 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:55:57.893871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:55:57.894120 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:55:57.894722 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:55:57.894763 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:55:57.896353 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:55:57.898009 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:55:57.898013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:55:57.898021 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:55:57.898247 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:55:57.898440 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:55:57.898442 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:55:57.898443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:55:57.898516 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:55:57.898870 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:55:57.898876 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:55:57.899129 (+ 253us) client_negotiation.cc:769] Sending connection context
0501 19:55:57.899292 (+ 163us) client_negotiation.cc:240] Negotiation successful
0501 19:55:57.899497 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":115,"threads_started":1}
I20240501 19:55:57.901348 9108 heartbeater.cc:347] Connected to a master server at 127.8.168.126:33165
I20240501 19:55:57.901564 9108 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.902047 9108 heartbeater.cc:511] Master 127.8.168.126:33165 requested a full tablet report, sending...
I20240501 19:55:57.903515 8917 ts_manager.cc:194] Registered new tserver with Master: a62095df3a3b453d94a00dca0bda7e72 (127.8.168.65:38293)
I20240501 19:55:57.904745 8917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:38111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
I20240501 19:55:57.914016 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33165
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:55:58.020202 9113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:58.020426 9113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:58.020491 9113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:58.023466 9113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:55:58.023532 9113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:58.023625 9113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:55:58.027621 9113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33165
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:58.028429 9113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:58.029318 9113 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:58.037617 9118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.037673 9119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.038234 9121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:58.038702 9113 server_base.cc:1034] running on GCE node
I20240501 19:55:58.039045 9113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:58.039577 9113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:58.040761 9113 hybrid_clock.cc:648] HybridClock initialized: now 1714593358040749 us; error 29 us; skew 500 ppm
May 01 19:55:58 dist-test-slave-tzct krb5kdc[8872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593358, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:55:58.043427 9113 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 19:55:58.044271 9113 webserver.cc:460] Webserver started at http://127.8.168.66:46431/ using document root <none> and password file <none>
I20240501 19:55:58.044790 9113 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:58.044845 9113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:58.044993 9113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:58.046681 9113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "805623aee52c437389f7890c9a0d4efb"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-tzct"
server_key: "735c0ea725f3049a38be18a69dfde157"
server_key_iv: "057c9002e41b56033792f2902bc32849"
server_key_version: "encryptionkey@0"
I20240501 19:55:58.047098 9113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "805623aee52c437389f7890c9a0d4efb"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-tzct"
server_key: "735c0ea725f3049a38be18a69dfde157"
server_key_iv: "057c9002e41b56033792f2902bc32849"
server_key_version: "encryptionkey@0"
I20240501 19:55:58.050251 9113 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:55:58.052340 9128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.053378 9113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:58.053468 9113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "805623aee52c437389f7890c9a0d4efb"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-tzct"
server_key: "735c0ea725f3049a38be18a69dfde157"
server_key_iv: "057c9002e41b56033792f2902bc32849"
server_key_version: "encryptionkey@0"
I20240501 19:55:58.053543 9113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:58.071945 9113 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:58.075161 9113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:58.075311 9113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:58.075834 9113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:58.076627 9113 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:58.076673 9113 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.076711 9113 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:58.076781 9113 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.085244 9113 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:45647
I20240501 19:55:58.085292 9241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:45647 every 8 connection(s)
I20240501 19:55:58.086155 9113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 19:55:58.089828 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 9113
I20240501 19:55:58.089955 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 19:55:58.090209 8865 external_mini_cluster.cc:1428] Setting key 5976248d0fd92eb01294328cb7d7cb7d
May 01 19:55:58 dist-test-slave-tzct krb5kdc[8872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593358, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.168.67@KRBTEST.COM; defaulting to no policy
I20240501 19:55:58.098657 8956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:55:58.087861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:59403 (local address 127.8.168.126:33165)
0501 19:55:58.088019 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0501 19:55:58.088024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:55:58.088616 (+ 592us) server_negotiation.cc:407] Connection header received
0501 19:55:58.089408 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:55:58.089412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:55:58.089460 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:55:58.089536 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:55:58.091462 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:55:58.092007 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:55:58.092819 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:55:58.092954 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:55:58.095119 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:55:58.095140 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:55:58.095144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:55:58.095170 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:55:58.096841 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:55:58.097312 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:55:58.097316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:55:58.097318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:55:58.097377 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:55:58.097623 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:55:58.097626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:55:58.097628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:55:58.097809 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:55:58.097913 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0501 19:55:58.098379 (+ 466us) server_negotiation.cc:299] Negotiation successful
0501 19:55:58.098490 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20240501 19:55:58.099278 9244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:55:58.088056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33165 (local address 127.8.168.66:59403)
0501 19:55:58.088479 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0501 19:55:58.088516 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0501 19:55:58.089229 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:55:58.089705 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:55:58.089715 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:55:58.090252 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:55:58.091315 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:55:58.091329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:55:58.092130 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:55:58.092135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:55:58.092688 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:55:58.092699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:55:58.092943 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:55:58.093611 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:55:58.093636 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:55:58.094970 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:55:58.096951 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:55:58.096956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:55:58.096964 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:55:58.097210 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:55:58.097466 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:55:58.097468 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:55:58.097470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:55:58.097543 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:55:58.097906 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:55:58.097911 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:55:58.098132 (+ 221us) client_negotiation.cc:769] Sending connection context
0501 19:55:58.098300 (+ 168us) client_negotiation.cc:240] Negotiation successful
0501 19:55:58.098511 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20240501 19:55:58.100397 9242 heartbeater.cc:347] Connected to a master server at 127.8.168.126:33165
I20240501 19:55:58.100622 9242 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.67@KRBTEST.COM" created.
I20240501 19:55:58.101277 9242 heartbeater.cc:511] Master 127.8.168.126:33165 requested a full tablet report, sending...
I20240501 19:55:58.102401 8917 ts_manager.cc:194] Registered new tserver with Master: 805623aee52c437389f7890c9a0d4efb (127.8.168.66:45647)
I20240501 19:55:58.103098 8917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:59403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
I20240501 19:55:58.116148 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33165
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:55:58.221004 9247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:58.221218 9247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:58.221275 9247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:58.224198 9247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:55:58.224256 9247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:58.224331 9247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:55:58.228358 9247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33165
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:58.229214 9247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:58.229990 9247 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:58.238121 9252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.238440 9253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.238982 9255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:58.239413 9247 server_base.cc:1034] running on GCE node
I20240501 19:55:58.239804 9247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:58.240470 9247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:58.241595 9247 hybrid_clock.cc:648] HybridClock initialized: now 1714593358241556 us; error 54 us; skew 500 ppm
May 01 19:55:58 dist-test-slave-tzct krb5kdc[8872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593358, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:55:58.244087 9247 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 19:55:58.244961 9247 webserver.cc:460] Webserver started at http://127.8.168.67:36247/ using document root <none> and password file <none>
I20240501 19:55:58.245460 9247 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:58.245514 9247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:58.245677 9247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:58.247391 9247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "c8cde15deec24a89870360acc9c9aced"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-tzct"
server_key: "32b1b29a3a3d6bba645cccdf3b663416"
server_key_iv: "e7059dde771043a6a4137eaaff3e25e2"
server_key_version: "encryptionkey@0"
I20240501 19:55:58.247866 9247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "c8cde15deec24a89870360acc9c9aced"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-tzct"
server_key: "32b1b29a3a3d6bba645cccdf3b663416"
server_key_iv: "e7059dde771043a6a4137eaaff3e25e2"
server_key_version: "encryptionkey@0"
I20240501 19:55:58.251065 9247 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:55:58.253276 9262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.254302 9247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:58.254401 9247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "c8cde15deec24a89870360acc9c9aced"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-tzct"
server_key: "32b1b29a3a3d6bba645cccdf3b663416"
server_key_iv: "e7059dde771043a6a4137eaaff3e25e2"
server_key_version: "encryptionkey@0"
I20240501 19:55:58.254482 9247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:58.277503 9247 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:58.280062 9247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:58.280218 9247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:58.280754 9247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:58.281545 9247 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:58.281592 9247 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.281630 9247 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:58.281643 9247 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.289784 9247 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:43821
I20240501 19:55:58.289821 9375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:43821 every 8 connection(s)
I20240501 19:55:58.290664 9247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 19:55:58.291111 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 9247
I20240501 19:55:58.291203 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 19:55:58.291410 8865 external_mini_cluster.cc:1428] Setting key 189b98b0101741904e76e6f5114c1e3c
May 01 19:55:58 dist-test-slave-tzct krb5kdc[8872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593358, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:55:58.302124 8956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:55:58.292284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:40821 (local address 127.8.168.126:33165)
0501 19:55:58.292409 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0501 19:55:58.292425 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0501 19:55:58.293213 (+ 788us) server_negotiation.cc:407] Connection header received
0501 19:55:58.294081 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:55:58.294084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:55:58.294131 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:55:58.294195 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:55:58.295727 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:55:58.296219 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:55:58.296827 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:55:58.296956 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:55:58.299027 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:55:58.299045 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:55:58.299048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:55:58.299074 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:55:58.300358 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:55:58.300895 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:55:58.300897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:55:58.300899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:55:58.300942 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:55:58.301147 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:55:58.301148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:55:58.301150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:55:58.301292 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:55:58.301371 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0501 19:55:58.301896 (+ 525us) server_negotiation.cc:299] Negotiation successful
0501 19:55:58.301993 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20240501 19:55:58.302922 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:55:58.292617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33165 (local address 127.8.168.67:40821)
0501 19:55:58.293073 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0501 19:55:58.293117 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0501 19:55:58.293869 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:55:58.294325 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:55:58.294333 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:55:58.294702 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:55:58.295598 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:55:58.295613 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:55:58.296305 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:55:58.296308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:55:58.296743 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:55:58.296751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:55:58.296910 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:55:58.297473 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:55:58.297497 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:55:58.298885 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:55:58.300483 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:55:58.300487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:55:58.300496 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:55:58.300810 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:55:58.301004 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:55:58.301006 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:55:58.301007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:55:58.301083 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:55:58.301401 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:55:58.301406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:55:58.301644 (+ 238us) client_negotiation.cc:769] Sending connection context
0501 19:55:58.301860 (+ 216us) client_negotiation.cc:240] Negotiation successful
0501 19:55:58.302087 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":142,"threads_started":1}
I20240501 19:55:58.304004 9376 heartbeater.cc:347] Connected to a master server at 127.8.168.126:33165
I20240501 19:55:58.304235 9376 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:58.304881 9376 heartbeater.cc:511] Master 127.8.168.126:33165 requested a full tablet report, sending...
I20240501 19:55:58.305814 8917 ts_manager.cc:194] Registered new tserver with Master: c8cde15deec24a89870360acc9c9aced (127.8.168.67:43821)
I20240501 19:55:58.306356 8917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:40821
I20240501 19:55:58.317857 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 01 19:55:58 dist-test-slave-tzct krb5kdc[8872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593358, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 01 19:55:58 dist-test-slave-tzct krb5kdc[8872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593358, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.168.65@KRBTEST.COM
I20240501 19:55:58.340094 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:55:58.332280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52250 (local address 127.8.168.65:38293)
0501 19:55:58.332642 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0501 19:55:58.332648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 19:55:58.332673 (+ 25us) server_negotiation.cc:407] Connection header received
0501 19:55:58.332754 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:55:58.332761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:55:58.332941 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:55:58.333064 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:55:58.333784 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:55:58.334475 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:55:58.335134 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:55:58.335297 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:55:58.337293 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:55:58.337318 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:55:58.337330 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:55:58.337354 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:55:58.338744 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:55:58.339080 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:55:58.339084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:55:58.339086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:55:58.339133 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:55:58.339276 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:55:58.339278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:55:58.339279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:55:58.339450 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:55:58.339592 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0501 19:55:58.339777 (+ 185us) server_negotiation.cc:299] Negotiation successful
0501 19:55:58.339916 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":154,"threads_started":1}
W20240501 19:55:58.341058 9022 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52250
May 01 19:55:58 dist-test-slave-tzct krb5kdc[8872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 01 19:55:58 dist-test-slave-tzct krb5kdc[8872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.65@KRBTEST.COM
I20240501 19:55:58.363374 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:55:58.356080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52260 (local address 127.8.168.65:38293)
0501 19:55:58.356220 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0501 19:55:58.356223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:55:58.356293 (+ 70us) server_negotiation.cc:407] Connection header received
0501 19:55:58.356419 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:55:58.356422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:55:58.356480 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:55:58.356555 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:55:58.357359 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:55:58.357870 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:55:58.358567 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:55:58.358758 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:55:58.360770 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:55:58.360787 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:55:58.360790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:55:58.360814 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:55:58.362232 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:55:58.362565 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:55:58.362568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:55:58.362569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:55:58.362609 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:55:58.362762 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:55:58.362764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:55:58.362765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:55:58.362882 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:55:58.362977 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0501 19:55:58.363122 (+ 145us) server_negotiation.cc:299] Negotiation successful
0501 19:55:58.363209 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20240501 19:55:58.364290 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 8979
I20240501 19:55:58.369769 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 9113
I20240501 19:55:58.376084 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 9247
I20240501 19:55:58.380625 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 8885
2024-05-01T19:55:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4498 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 19:55:58 dist-test-slave-tzct krb5kdc[9390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 19:55:58 dist-test-slave-tzct krb5kdc[9390](info): set up 2 sockets
May 01 19:55:58 dist-test-slave-tzct krb5kdc[9390](info): commencing operation
krb5kdc: starting...
W20240501 19:55:59.276705 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.876s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 19:55:59 dist-test-slave-tzct krb5kdc[9390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T19:55:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:55:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 19:55:59.395560 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:44489
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:43157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:44489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:55:59.501603 9403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:59.501844 9403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:59.501895 9403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:59.504910 9403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:55:59.504967 9403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:55:59.504992 9403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:59.505009 9403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:55:59.505025 9403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:55:59.509166 9403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:43157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:44489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:44489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:59.510010 9403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:59.510872 9403 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:59.518250 9411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.518529 9408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.518944 9409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:59.519608 9403 server_base.cc:1034] running on GCE node
I20240501 19:55:59.520093 9403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:59.521018 9403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:59.522194 9403 hybrid_clock.cc:648] HybridClock initialized: now 1714593359522179 us; error 30 us; skew 500 ppm
May 01 19:55:59 dist-test-slave-tzct krb5kdc[9390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593359, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:55:59.524979 9403 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 19:55:59.525925 9403 webserver.cc:460] Webserver started at http://127.8.168.126:36721/ using document root <none> and password file <none>
I20240501 19:55:59.526460 9403 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:59.526520 9403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:59.526696 9403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:59.528512 9403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "68f53c14b7e34d8bb9ca98c1eaba256b"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-tzct"
server_key: "74d9420de3b0a52a99ef8a05883571d5"
server_key_iv: "9b4dc1efa0a18d403a0db1cc40106714"
server_key_version: "encryptionkey@0"
I20240501 19:55:59.529013 9403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "68f53c14b7e34d8bb9ca98c1eaba256b"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-tzct"
server_key: "74d9420de3b0a52a99ef8a05883571d5"
server_key_iv: "9b4dc1efa0a18d403a0db1cc40106714"
server_key_version: "encryptionkey@0"
I20240501 19:55:59.532542 9403 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 19:55:59.534894 9418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.535991 9403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:59.536108 9403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "68f53c14b7e34d8bb9ca98c1eaba256b"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-tzct"
server_key: "74d9420de3b0a52a99ef8a05883571d5"
server_key_iv: "9b4dc1efa0a18d403a0db1cc40106714"
server_key_version: "encryptionkey@0"
I20240501 19:55:59.536207 9403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:59.553373 9403 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:59.556480 9403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:59.556654 9403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:59.563223 9403 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:44489
I20240501 19:55:59.563261 9470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:44489 every 8 connection(s)
I20240501 19:55:59.564199 9403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:55:59.567025 9471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:59.571396 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 9403
I20240501 19:55:59.571534 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:55:59.571799 8865 external_mini_cluster.cc:1428] Setting key 5ef36827c99a8f00b3c5a02fa21f5bff
I20240501 19:55:59.572197 9471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b: Bootstrap starting.
I20240501 19:55:59.574673 9471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:59.575520 9471 log.cc:824] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:59.577504 9471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b: No bootstrap required, opened a new log
May 01 19:55:59 dist-test-slave-tzct krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:55:59.581154 9471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f53c14b7e34d8bb9ca98c1eaba256b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 44489 } }
I20240501 19:55:59.581387 9471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:59.581459 9471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f53c14b7e34d8bb9ca98c1eaba256b, State: Initialized, Role: FOLLOWER
I20240501 19:55:59.581948 9471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f53c14b7e34d8bb9ca98c1eaba256b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 44489 } }
I20240501 19:55:59.582108 9471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:55:59.582173 9471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:55:59.582289 9471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:59.583281 9471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f53c14b7e34d8bb9ca98c1eaba256b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 44489 } }
I20240501 19:55:59.583628 9471 leader_election.cc:302] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68f53c14b7e34d8bb9ca98c1eaba256b; no voters:
I20240501 19:55:59.583941 9471 leader_election.cc:288] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:55:59.584146 9476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:59.584378 9476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [term 1 LEADER]: Becoming Leader. State: Replica: 68f53c14b7e34d8bb9ca98c1eaba256b, State: Running, Role: LEADER
I20240501 19:55:59.584700 9476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f53c14b7e34d8bb9ca98c1eaba256b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 44489 } }
I20240501 19:55:59.584905 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:55:59.573140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42216 (local address 127.8.168.126:44489)
0501 19:55:59.573554 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0501 19:55:59.573566 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 19:55:59.573607 (+ 41us) server_negotiation.cc:407] Connection header received
0501 19:55:59.574421 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:55:59.574448 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:55:59.574828 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:55:59.575228 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:55:59.575973 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:55:59.577082 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:55:59.577729 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:55:59.577997 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:55:59.580152 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:55:59.580176 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:55:59.580189 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:55:59.580252 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:55:59.582289 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:55:59.582688 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:55:59.582693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:55:59.582699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:55:59.582788 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:55:59.582990 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:55:59.582993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:55:59.582994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:55:59.583441 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:55:59.583649 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0501 19:55:59.583830 (+ 181us) server_negotiation.cc:299] Negotiation successful
0501 19:55:59.584108 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":116,"threads_started":1}
I20240501 19:55:59.585116 9471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:55:59.586222 9478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68f53c14b7e34d8bb9ca98c1eaba256b. Latest consensus state: current_term: 1 leader_uuid: "68f53c14b7e34d8bb9ca98c1eaba256b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f53c14b7e34d8bb9ca98c1eaba256b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 44489 } } }
I20240501 19:55:59.586218 9477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68f53c14b7e34d8bb9ca98c1eaba256b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f53c14b7e34d8bb9ca98c1eaba256b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 44489 } } }
I20240501 19:55:59.586347 9478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:59.586360 9477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:59.586809 9481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:55:59.589735 9481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:55:59.594916 9481 catalog_manager.cc:1347] Generated new cluster ID: c83d865a160b43d5832a1afd117896a5
I20240501 19:55:59.594985 9481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:55:59.609694 9481 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:55:59.610939 9481 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:55:59.618872 9481 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 68f53c14b7e34d8bb9ca98c1eaba256b: Generated new TSK 0
I20240501 19:55:59.619683 9481 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
I20240501 19:55:59.651363 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:44489
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:43157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:55:59.757457 9497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:59.757695 9497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:59.757778 9497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:59.760850 9497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:55:59.760924 9497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:59.761025 9497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:55:59.765612 9497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:43157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:44489
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:59.766781 9497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:59.767899 9497 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:59.777319 9505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.777670 9502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.778246 9503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:59.778764 9497 server_base.cc:1034] running on GCE node
I20240501 19:55:59.779232 9497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:59.779989 9497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:59.781193 9497 hybrid_clock.cc:648] HybridClock initialized: now 1714593359781156 us; error 69 us; skew 500 ppm
May 01 19:55:59 dist-test-slave-tzct krb5kdc[9390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593359, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:55:59.784741 9497 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 19:55:59.786005 9497 webserver.cc:460] Webserver started at http://127.8.168.65:44951/ using document root <none> and password file <none>
I20240501 19:55:59.786765 9497 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:59.786866 9497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:59.787098 9497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:59.789482 9497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "5acb9190154e4308bc0ec4293382e4dd"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-tzct"
server_key: "c0e095f632b85580340305d2c5d8dc20"
server_key_iv: "c4adfbfe4650a5ad15e73ae55da79500"
server_key_version: "encryptionkey@0"
I20240501 19:55:59.790108 9497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "5acb9190154e4308bc0ec4293382e4dd"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-tzct"
server_key: "c0e095f632b85580340305d2c5d8dc20"
server_key_iv: "c4adfbfe4650a5ad15e73ae55da79500"
server_key_version: "encryptionkey@0"
I20240501 19:55:59.794423 9497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 19:55:59.797497 9512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.798938 9497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:55:59.799060 9497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "5acb9190154e4308bc0ec4293382e4dd"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-tzct"
server_key: "c0e095f632b85580340305d2c5d8dc20"
server_key_iv: "c4adfbfe4650a5ad15e73ae55da79500"
server_key_version: "encryptionkey@0"
I20240501 19:55:59.799177 9497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:59.832048 9497 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:59.834651 9497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:59.834787 9497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:59.835278 9497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:59.836045 9497 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:59.836089 9497 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.836127 9497 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:59.836148 9497 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.844578 9497 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:44441
I20240501 19:55:59.844619 9625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:44441 every 8 connection(s)
I20240501 19:55:59.845507 9497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 19:55:59.847640 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 9497
I20240501 19:55:59.847777 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 19:55:59.848042 8865 external_mini_cluster.cc:1428] Setting key eacabfdc18927faa1e292ff8eff2f60a
May 01 19:55:59 dist-test-slave-tzct krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593359, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.168.66@KRBTEST.COM; defaulting to no policy
I20240501 19:55:59.857149 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:55:59.847104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:33957 (local address 127.8.168.126:44489)
0501 19:55:59.847223 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0501 19:55:59.847226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:55:59.847946 (+ 720us) server_negotiation.cc:407] Connection header received
0501 19:55:59.848821 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:55:59.848825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:55:59.848900 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:55:59.848975 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:55:59.850606 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:55:59.851114 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:55:59.851723 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:55:59.851908 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:55:59.854152 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:55:59.854170 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:55:59.854172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:55:59.854197 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:55:59.855465 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:55:59.855951 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:55:59.855953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:55:59.855954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:55:59.855993 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:55:59.856195 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:55:59.856197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:55:59.856198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:55:59.856333 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:55:59.856398 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0501 19:55:59.856907 (+ 509us) server_negotiation.cc:299] Negotiation successful
0501 19:55:59.857021 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240501 19:55:59.857754 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:55:59.847353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:44489 (local address 127.8.168.65:33957)
0501 19:55:59.847807 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0501 19:55:59.847853 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0501 19:55:59.848598 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:55:59.849149 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:55:59.849160 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:55:59.849568 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:55:59.850470 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:55:59.850484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:55:59.851223 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:55:59.851226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:55:59.851617 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:55:59.851624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:55:59.851827 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:55:59.852411 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:55:59.852453 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:55:59.853978 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:55:59.855585 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:55:59.855591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:55:59.855602 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:55:59.855871 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:55:59.856060 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:55:59.856062 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:55:59.856063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:55:59.856137 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:55:59.856413 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:55:59.856417 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:55:59.856657 (+ 240us) client_negotiation.cc:769] Sending connection context
0501 19:55:59.856854 (+ 197us) client_negotiation.cc:240] Negotiation successful
0501 19:55:59.857079 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.66@KRBTEST.COM" created.
I20240501 19:55:59.858879 9626 heartbeater.cc:347] Connected to a master server at 127.8.168.126:44489
I20240501 19:55:59.859125 9626 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:59.859671 9626 heartbeater.cc:511] Master 127.8.168.126:44489 requested a full tablet report, sending...
I20240501 19:55:59.861076 9433 ts_manager.cc:194] Registered new tserver with Master: 5acb9190154e4308bc0ec4293382e4dd (127.8.168.65:44441)
I20240501 19:55:59.862187 9433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:33957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
I20240501 19:55:59.872277 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:44489
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:43157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:55:59.976260 9631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:59.976473 9631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:59.976529 9631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:59.979521 9631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:55:59.979584 9631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:59.979662 9631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:55:59.983731 9631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:43157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:44489
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:59.984558 9631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:59.985419 9631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:59.993068 9639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.993561 9636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.994079 9637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:59.994678 9631 server_base.cc:1034] running on GCE node
I20240501 19:55:59.995025 9631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:59.995589 9631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:59.996760 9631 hybrid_clock.cc:648] HybridClock initialized: now 1714593359996742 us; error 35 us; skew 500 ppm
May 01 19:55:59 dist-test-slave-tzct krb5kdc[9390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593359, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:55:59.999389 9631 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 19:56:00.000264 9631 webserver.cc:460] Webserver started at http://127.8.168.66:41063/ using document root <none> and password file <none>
I20240501 19:56:00.000833 9631 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:00.000896 9631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:00.001048 9631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:00.002779 9631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "0dc2612096264b07a015817868d0435c"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-tzct"
server_key: "17595459409734d214db64ba068b78a0"
server_key_iv: "6bd0fa022b727e27f5c26edf3b014289"
server_key_version: "encryptionkey@0"
I20240501 19:56:00.003207 9631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "0dc2612096264b07a015817868d0435c"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-tzct"
server_key: "17595459409734d214db64ba068b78a0"
server_key_iv: "6bd0fa022b727e27f5c26edf3b014289"
server_key_version: "encryptionkey@0"
I20240501 19:56:00.006547 9631 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:00.008666 9646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.009774 9631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:00.009883 9631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "0dc2612096264b07a015817868d0435c"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-tzct"
server_key: "17595459409734d214db64ba068b78a0"
server_key_iv: "6bd0fa022b727e27f5c26edf3b014289"
server_key_version: "encryptionkey@0"
I20240501 19:56:00.009984 9631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:00.025411 9631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:00.028129 9631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:00.028283 9631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:00.028878 9631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:00.029656 9631 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:00.029701 9631 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.029745 9631 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:00.029785 9631 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.038364 9631 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:37723
I20240501 19:56:00.038403 9759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:37723 every 8 connection(s)
I20240501 19:56:00.039294 9631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
May 01 19:56:00 dist-test-slave-tzct krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593359, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:00.047721 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 9631
I20240501 19:56:00.047847 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 19:56:00.048107 8865 external_mini_cluster.cc:1428] Setting key 3d737e736abd1ef83ef14e902ca1528a
I20240501 19:56:00.051920 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.041059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:38879 (local address 127.8.168.126:44489)
0501 19:56:00.041205 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:00.041208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:00.041916 (+ 708us) server_negotiation.cc:407] Connection header received
0501 19:56:00.042712 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:00.042715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:00.042774 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:00.042854 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:00.044418 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.044949 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.045565 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.045746 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.048067 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:00.048089 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:00.048091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:00.048122 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:00.049840 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:00.050369 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:00.050373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:00.050375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:00.050430 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:00.050797 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:00.050799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:00.050801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:00.050975 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:00.051066 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:00.051596 (+ 530us) server_negotiation.cc:299] Negotiation successful
0501 19:56:00.051744 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20240501 19:56:00.052598 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.041375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:44489 (local address 127.8.168.66:38879)
0501 19:56:00.041792 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:00.041827 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:00.042512 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:00.043021 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:00.043029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:00.043383 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:00.044272 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:00.044287 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.045061 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.045064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:00.045456 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:00.045463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.045677 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.046309 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:00.046335 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:00.047886 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:00.049973 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:00.049977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:00.049987 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:00.050260 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:00.050550 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:00.050553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:00.050555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:00.050663 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:00.051085 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:00.051090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:00.051349 (+ 259us) client_negotiation.cc:769] Sending connection context
0501 19:56:00.051526 (+ 177us) client_negotiation.cc:240] Negotiation successful
0501 19:56:00.051804 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
I20240501 19:56:00.053789 9760 heartbeater.cc:347] Connected to a master server at 127.8.168.126:44489
I20240501 19:56:00.054044 9760 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:00.054716 9760 heartbeater.cc:511] Master 127.8.168.126:44489 requested a full tablet report, sending...
I20240501 19:56:00.055959 9433 ts_manager.cc:194] Registered new tserver with Master: 0dc2612096264b07a015817868d0435c (127.8.168.66:37723)
I20240501 19:56:00.056757 9433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:38879
WARNING: no policy specified for kudu/127.8.168.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
I20240501 19:56:00.075556 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:44489
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:43157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:00.181442 9765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:00.181674 9765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:00.181757 9765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:00.184747 9765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:00.184811 9765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:00.184904 9765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:56:00.189006 9765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:43157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:44489
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:00.189898 9765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:00.190753 9765 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:00.198681 9773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:00.199160 9771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:00.199826 9770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:00.200248 9765 server_base.cc:1034] running on GCE node
I20240501 19:56:00.200603 9765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:00.201166 9765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:00.202317 9765 hybrid_clock.cc:648] HybridClock initialized: now 1714593360202305 us; error 25 us; skew 500 ppm
May 01 19:56:00 dist-test-slave-tzct krb5kdc[9390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593360, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:00.204949 9765 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 19:56:00.205826 9765 webserver.cc:460] Webserver started at http://127.8.168.67:38511/ using document root <none> and password file <none>
I20240501 19:56:00.206347 9765 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:00.206400 9765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:00.206568 9765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:00.208321 9765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "a786a66433c14b4db0cd1485df4ba85e"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-tzct"
server_key: "f7cfc7f374dda0531979b5cca0640010"
server_key_iv: "196399ff0b612bb29bbf76c5a8f74a56"
server_key_version: "encryptionkey@0"
I20240501 19:56:00.208818 9765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "a786a66433c14b4db0cd1485df4ba85e"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-tzct"
server_key: "f7cfc7f374dda0531979b5cca0640010"
server_key_iv: "196399ff0b612bb29bbf76c5a8f74a56"
server_key_version: "encryptionkey@0"
I20240501 19:56:00.211966 9765 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 19:56:00.214198 9780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.215266 9765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:00.215380 9765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "a786a66433c14b4db0cd1485df4ba85e"
format_stamp: "Formatted at 2024-05-01 19:56:00 on dist-test-slave-tzct"
server_key: "f7cfc7f374dda0531979b5cca0640010"
server_key_iv: "196399ff0b612bb29bbf76c5a8f74a56"
server_key_version: "encryptionkey@0"
I20240501 19:56:00.215466 9765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:00.231119 9765 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:00.234326 9765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:00.234513 9765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:00.235174 9765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:00.236163 9765 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:00.236235 9765 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.236299 9765 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:00.236330 9765 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:00.246307 9765 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:38647
I20240501 19:56:00.246322 9893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:38647 every 8 connection(s)
I20240501 19:56:00.247208 9765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 19:56:00.251055 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 9765
I20240501 19:56:00.251145 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 19:56:00.251358 8865 external_mini_cluster.cc:1428] Setting key dde5edd95ef78a7933539fe68a4e2a3a
May 01 19:56:00 dist-test-slave-tzct krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593360, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:00.258857 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.248814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:40271 (local address 127.8.168.126:44489)
0501 19:56:00.248954 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:00.248958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:00.249585 (+ 627us) server_negotiation.cc:407] Connection header received
0501 19:56:00.250356 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:00.250359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:00.250415 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:00.250489 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:00.252302 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.253097 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.253803 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.253942 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.255910 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:00.255926 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:00.255928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:00.255953 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:00.257303 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:00.257737 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:00.257739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:00.257761 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:00.257798 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:00.257996 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:00.257999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:00.258000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:00.258134 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:00.258206 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:00.258612 (+ 406us) server_negotiation.cc:299] Negotiation successful
0501 19:56:00.258715 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240501 19:56:00.259410 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.249071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:44489 (local address 127.8.168.67:40271)
0501 19:56:00.249462 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:00.249495 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:00.250158 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:00.250649 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:00.250657 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:00.251014 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:00.252118 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:00.252138 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.253234 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.253237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:00.253665 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:00.253673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.253876 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.254423 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:00.254448 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:00.255779 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:00.257415 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:00.257420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:00.257428 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:00.257653 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:00.257862 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:00.257864 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:00.257865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:00.257933 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:00.258206 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:00.258210 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:00.258431 (+ 221us) client_negotiation.cc:769] Sending connection context
0501 19:56:00.258583 (+ 152us) client_negotiation.cc:240] Negotiation successful
0501 19:56:00.258783 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":97,"threads_started":1}
I20240501 19:56:00.260362 9894 heartbeater.cc:347] Connected to a master server at 127.8.168.126:44489
I20240501 19:56:00.260567 9894 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:00.261101 9894 heartbeater.cc:511] Master 127.8.168.126:44489 requested a full tablet report, sending...
I20240501 19:56:00.262029 9433 ts_manager.cc:194] Registered new tserver with Master: a786a66433c14b4db0cd1485df4ba85e (127.8.168.67:38647)
I20240501 19:56:00.262523 9433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:40271
I20240501 19:56:00.264811 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 19:56:00.272820 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.267187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.8.168.126:44489)
0501 19:56:00.267304 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:00.267307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:00.267399 (+ 92us) server_negotiation.cc:407] Connection header received
0501 19:56:00.267503 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:00.267505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:00.267541 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:00.267605 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:00.268277 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.268772 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.269427 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.269559 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.270455 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:00.270469 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:00.270471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:00.270491 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:00.271727 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:00.272012 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:00.272015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:00.272016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:00.272051 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:00.272184 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:00.272185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:00.272186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:00.272330 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:00.272418 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:00.272554 (+ 136us) server_negotiation.cc:299] Negotiation successful
0501 19:56:00.272638 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240501 19:56:00.276377 9433 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:56:00.278808 9433 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:00.290899 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.286402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:44441 (local address 127.0.0.1:35538)
0501 19:56:00.286721 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:00.286757 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:00.286890 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:00.288124 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:00.288130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:00.288149 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:00.288436 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:00.288445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.289632 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.289635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:00.290399 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:00.290408 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.290521 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.290579 (+ 58us) client_negotiation.cc:769] Sending connection context
0501 19:56:00.290669 (+ 90us) client_negotiation.cc:240] Negotiation successful
0501 19:56:00.290752 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":86,"threads_started":1}
I20240501 19:56:00.291338 9907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.287059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:38647 (local address 127.0.0.1:39164)
0501 19:56:00.287274 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:00.287287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:00.287367 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:00.288559 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:00.288562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:00.288580 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:00.288837 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:00.288846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.290196 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.290199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:00.291012 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:00.291022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.291135 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.291151 (+ 16us) client_negotiation.cc:769] Sending connection context
0501 19:56:00.291198 (+ 47us) client_negotiation.cc:240] Negotiation successful
0501 19:56:00.291247 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":92,"threads_started":1}
I20240501 19:56:00.291481 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.287349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.8.168.65:44441)
0501 19:56:00.287644 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:00.287650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:00.287673 (+ 23us) server_negotiation.cc:407] Connection header received
0501 19:56:00.287802 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:00.287808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:00.287946 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:00.288090 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:00.288549 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.289529 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.290531 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.291063 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.291183 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:00.291252 (+ 69us) server_negotiation.cc:299] Negotiation successful
0501 19:56:00.291332 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":117,"threads_started":1}
I20240501 19:56:00.291793 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.286487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:37723 (local address 127.0.0.1:39946)
0501 19:56:00.286743 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:00.286759 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:00.286892 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:00.288801 (+ 1909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:00.288804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:00.288823 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:00.289037 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:00.289044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.290313 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.290316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:00.291435 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:00.291446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.291550 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.291566 (+ 16us) client_negotiation.cc:769] Sending connection context
0501 19:56:00.291611 (+ 45us) client_negotiation.cc:240] Negotiation successful
0501 19:56:00.291668 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"spinlock_wait_cycles":59776,"thread_start_us":112,"threads_started":1}
I20240501 19:56:00.292631 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.287243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39164 (local address 127.8.168.67:38647)
0501 19:56:00.287658 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:00.287664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:00.287682 (+ 18us) server_negotiation.cc:407] Connection header received
0501 19:56:00.287754 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:00.287757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:00.287933 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:00.288094 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:00.288954 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.290069 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.291725 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.292213 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.292325 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:00.292389 (+ 64us) server_negotiation.cc:299] Negotiation successful
0501 19:56:00.292498 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":141,"threads_started":1}
I20240501 19:56:00.292763 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.287255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39946 (local address 127.8.168.66:37723)
0501 19:56:00.288392 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:00.288398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:00.288417 (+ 19us) server_negotiation.cc:407] Connection header received
0501 19:56:00.288496 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:00.288501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:00.288649 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:00.288814 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:00.289153 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.290208 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.291926 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.292338 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.292461 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:00.292527 (+ 66us) server_negotiation.cc:299] Negotiation successful
0501 19:56:00.292602 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1029,"thread_start_us":113,"threads_started":1}
I20240501 19:56:00.294298 9560 tablet_service.cc:1467] Processing CreateTablet for tablet 35970e99b91b48bda92e6f5173c31826 (DEFAULT_TABLE table=test-table [id=11312b4e187e42faab95c4cc54b96c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:00.294561 9828 tablet_service.cc:1467] Processing CreateTablet for tablet 35970e99b91b48bda92e6f5173c31826 (DEFAULT_TABLE table=test-table [id=11312b4e187e42faab95c4cc54b96c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:00.294843 9692 tablet_service.cc:1467] Processing CreateTablet for tablet 35970e99b91b48bda92e6f5173c31826 (DEFAULT_TABLE table=test-table [id=11312b4e187e42faab95c4cc54b96c75]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:00.295302 9828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35970e99b91b48bda92e6f5173c31826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.295308 9560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35970e99b91b48bda92e6f5173c31826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.295576 9692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35970e99b91b48bda92e6f5173c31826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.300562 9911 tablet_bootstrap.cc:492] T 35970e99b91b48bda92e6f5173c31826 P a786a66433c14b4db0cd1485df4ba85e: Bootstrap starting.
I20240501 19:56:00.302024 9912 tablet_bootstrap.cc:492] T 35970e99b91b48bda92e6f5173c31826 P 5acb9190154e4308bc0ec4293382e4dd: Bootstrap starting.
I20240501 19:56:00.302611 9911 tablet_bootstrap.cc:654] T 35970e99b91b48bda92e6f5173c31826 P a786a66433c14b4db0cd1485df4ba85e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.303244 9911 log.cc:824] T 35970e99b91b48bda92e6f5173c31826 P a786a66433c14b4db0cd1485df4ba85e: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:00.303578 9912 tablet_bootstrap.cc:654] T 35970e99b91b48bda92e6f5173c31826 P 5acb9190154e4308bc0ec4293382e4dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.304271 9913 tablet_bootstrap.cc:492] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c: Bootstrap starting.
I20240501 19:56:00.304270 9912 log.cc:824] T 35970e99b91b48bda92e6f5173c31826 P 5acb9190154e4308bc0ec4293382e4dd: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:00.305102 9911 tablet_bootstrap.cc:492] T 35970e99b91b48bda92e6f5173c31826 P a786a66433c14b4db0cd1485df4ba85e: No bootstrap required, opened a new log
I20240501 19:56:00.305308 9911 ts_tablet_manager.cc:1367] T 35970e99b91b48bda92e6f5173c31826 P a786a66433c14b4db0cd1485df4ba85e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:00.306010 9912 tablet_bootstrap.cc:492] T 35970e99b91b48bda92e6f5173c31826 P 5acb9190154e4308bc0ec4293382e4dd: No bootstrap required, opened a new log
I20240501 19:56:00.306162 9912 ts_tablet_manager.cc:1367] T 35970e99b91b48bda92e6f5173c31826 P 5acb9190154e4308bc0ec4293382e4dd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:00.306296 9913 tablet_bootstrap.cc:654] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.307116 9913 log.cc:824] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:00.309028 9913 tablet_bootstrap.cc:492] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c: No bootstrap required, opened a new log
I20240501 19:56:00.309231 9913 ts_tablet_manager.cc:1367] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:00.308777 9911 raft_consensus.cc:348] T 35970e99b91b48bda92e6f5173c31826 P a786a66433c14b4db0cd1485df4ba85e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acb9190154e4308bc0ec4293382e4dd" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44441 } } peers { permanent_uuid: "0dc2612096264b07a015817868d0435c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37723 } } peers { permanent_uuid: "a786a66433c14b4db0cd1485df4ba85e" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 38647 } }
I20240501 19:56:00.309151 9912 raft_consensus.cc:348] T 35970e99b91b48bda92e6f5173c31826 P 5acb9190154e4308bc0ec4293382e4dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acb9190154e4308bc0ec4293382e4dd" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44441 } } peers { permanent_uuid: "0dc2612096264b07a015817868d0435c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37723 } } peers { permanent_uuid: "a786a66433c14b4db0cd1485df4ba85e" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 38647 } }
I20240501 19:56:00.309336 9912 raft_consensus.cc:374] T 35970e99b91b48bda92e6f5173c31826 P 5acb9190154e4308bc0ec4293382e4dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.309338 9911 raft_consensus.cc:374] T 35970e99b91b48bda92e6f5173c31826 P a786a66433c14b4db0cd1485df4ba85e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.309386 9912 raft_consensus.cc:729] T 35970e99b91b48bda92e6f5173c31826 P 5acb9190154e4308bc0ec4293382e4dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5acb9190154e4308bc0ec4293382e4dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.309391 9911 raft_consensus.cc:729] T 35970e99b91b48bda92e6f5173c31826 P a786a66433c14b4db0cd1485df4ba85e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a786a66433c14b4db0cd1485df4ba85e, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.309852 9912 consensus_queue.cc:260] T 35970e99b91b48bda92e6f5173c31826 P 5acb9190154e4308bc0ec4293382e4dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acb9190154e4308bc0ec4293382e4dd" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44441 } } peers { permanent_uuid: "0dc2612096264b07a015817868d0435c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37723 } } peers { permanent_uuid: "a786a66433c14b4db0cd1485df4ba85e" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 38647 } }
I20240501 19:56:00.309854 9911 consensus_queue.cc:260] T 35970e99b91b48bda92e6f5173c31826 P a786a66433c14b4db0cd1485df4ba85e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acb9190154e4308bc0ec4293382e4dd" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44441 } } peers { permanent_uuid: "0dc2612096264b07a015817868d0435c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37723 } } peers { permanent_uuid: "a786a66433c14b4db0cd1485df4ba85e" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 38647 } }
I20240501 19:56:00.310511 9626 heartbeater.cc:503] Master 127.8.168.126:44489 was elected leader, sending a full tablet report...
I20240501 19:56:00.310714 9912 ts_tablet_manager.cc:1398] T 35970e99b91b48bda92e6f5173c31826 P 5acb9190154e4308bc0ec4293382e4dd: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 19:56:00.310745 9894 heartbeater.cc:503] Master 127.8.168.126:44489 was elected leader, sending a full tablet report...
I20240501 19:56:00.312367 9911 ts_tablet_manager.cc:1398] T 35970e99b91b48bda92e6f5173c31826 P a786a66433c14b4db0cd1485df4ba85e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 19:56:00.312573 9913 raft_consensus.cc:348] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acb9190154e4308bc0ec4293382e4dd" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44441 } } peers { permanent_uuid: "0dc2612096264b07a015817868d0435c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37723 } } peers { permanent_uuid: "a786a66433c14b4db0cd1485df4ba85e" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 38647 } }
I20240501 19:56:00.312754 9913 raft_consensus.cc:374] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.312817 9913 raft_consensus.cc:729] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc2612096264b07a015817868d0435c, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.313182 9913 consensus_queue.cc:260] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acb9190154e4308bc0ec4293382e4dd" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44441 } } peers { permanent_uuid: "0dc2612096264b07a015817868d0435c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37723 } } peers { permanent_uuid: "a786a66433c14b4db0cd1485df4ba85e" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 38647 } }
I20240501 19:56:00.313937 9913 ts_tablet_manager.cc:1398] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 19:56:00.314391 9760 heartbeater.cc:503] Master 127.8.168.126:44489 was elected leader, sending a full tablet report...
W20240501 19:56:00.346583 9627 tablet.cc:2369] T 35970e99b91b48bda92e6f5173c31826 P 5acb9190154e4308bc0ec4293382e4dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:00.399638 9919 raft_consensus.cc:482] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:00.399849 9919 raft_consensus.cc:504] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acb9190154e4308bc0ec4293382e4dd" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44441 } } peers { permanent_uuid: "0dc2612096264b07a015817868d0435c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37723 } } peers { permanent_uuid: "a786a66433c14b4db0cd1485df4ba85e" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 38647 } }
I20240501 19:56:00.400964 9919 leader_election.cc:288] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5acb9190154e4308bc0ec4293382e4dd (127.8.168.65:44441), a786a66433c14b4db0cd1485df4ba85e (127.8.168.67:38647)
I20240501 19:56:00.404162 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.401221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:44441 (local address 127.8.168.66:60951)
0501 19:56:00.401381 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:00.401404 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:00.401525 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:00.401869 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:00.401871 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:00.401888 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:00.402071 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:00.402076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.403006 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.403009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:00.403809 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:00.403818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.403926 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.403947 (+ 21us) client_negotiation.cc:769] Sending connection context
0501 19:56:00.403989 (+ 42us) client_negotiation.cc:240] Negotiation successful
0501 19:56:00.404038 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20240501 19:56:00.404413 9920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.401491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:38647 (local address 127.8.168.66:52999)
0501 19:56:00.401860 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:00.401870 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:00.401953 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:00.402187 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:00.402190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:00.402203 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:00.402391 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:00.402395 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.403259 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.403262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:00.404081 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:00.404089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.404178 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.404194 (+ 16us) client_negotiation.cc:769] Sending connection context
0501 19:56:00.404238 (+ 44us) client_negotiation.cc:240] Negotiation successful
0501 19:56:00.404286 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":130,"threads_started":1}
I20240501 19:56:00.404649 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.401300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:60951 (local address 127.8.168.65:44441)
0501 19:56:00.401420 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:00.401425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:00.401463 (+ 38us) server_negotiation.cc:407] Connection header received
0501 19:56:00.401636 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:00.401640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:00.401694 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:00.401772 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:00.402175 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.402887 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.403925 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.404387 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.404427 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:00.404476 (+ 49us) server_negotiation.cc:299] Negotiation successful
0501 19:56:00.404529 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240501 19:56:00.404907 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.401628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:52999 (local address 127.8.168.67:38647)
0501 19:56:00.401784 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:00.401788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:00.401931 (+ 143us) server_negotiation.cc:407] Connection header received
0501 19:56:00.402042 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:00.402045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:00.402089 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:00.402155 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:00.402481 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.403138 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.404201 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.404635 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.404667 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:00.404764 (+ 97us) server_negotiation.cc:299] Negotiation successful
0501 19:56:00.404822 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20240501 19:56:00.405251 9580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35970e99b91b48bda92e6f5173c31826" candidate_uuid: "0dc2612096264b07a015817868d0435c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5acb9190154e4308bc0ec4293382e4dd" is_pre_election: true
I20240501 19:56:00.405407 9848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35970e99b91b48bda92e6f5173c31826" candidate_uuid: "0dc2612096264b07a015817868d0435c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a786a66433c14b4db0cd1485df4ba85e" is_pre_election: true
I20240501 19:56:00.405530 9580 raft_consensus.cc:2454] T 35970e99b91b48bda92e6f5173c31826 P 5acb9190154e4308bc0ec4293382e4dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dc2612096264b07a015817868d0435c in term 0.
I20240501 19:56:00.405633 9848 raft_consensus.cc:2454] T 35970e99b91b48bda92e6f5173c31826 P a786a66433c14b4db0cd1485df4ba85e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dc2612096264b07a015817868d0435c in term 0.
I20240501 19:56:00.405959 9650 leader_election.cc:302] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc2612096264b07a015817868d0435c, 5acb9190154e4308bc0ec4293382e4dd; no voters:
I20240501 19:56:00.406198 9919 raft_consensus.cc:2781] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:00.406265 9919 raft_consensus.cc:482] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:00.406311 9919 raft_consensus.cc:3037] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.407225 9919 raft_consensus.cc:504] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acb9190154e4308bc0ec4293382e4dd" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44441 } } peers { permanent_uuid: "0dc2612096264b07a015817868d0435c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37723 } } peers { permanent_uuid: "a786a66433c14b4db0cd1485df4ba85e" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 38647 } }
I20240501 19:56:00.407532 9919 leader_election.cc:288] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c [CANDIDATE]: Term 1 election: Requested vote from peers 5acb9190154e4308bc0ec4293382e4dd (127.8.168.65:44441), a786a66433c14b4db0cd1485df4ba85e (127.8.168.67:38647)
I20240501 19:56:00.407845 9580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35970e99b91b48bda92e6f5173c31826" candidate_uuid: "0dc2612096264b07a015817868d0435c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5acb9190154e4308bc0ec4293382e4dd"
I20240501 19:56:00.407922 9848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35970e99b91b48bda92e6f5173c31826" candidate_uuid: "0dc2612096264b07a015817868d0435c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a786a66433c14b4db0cd1485df4ba85e"
I20240501 19:56:00.407956 9580 raft_consensus.cc:3037] T 35970e99b91b48bda92e6f5173c31826 P 5acb9190154e4308bc0ec4293382e4dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.408031 9848 raft_consensus.cc:3037] T 35970e99b91b48bda92e6f5173c31826 P a786a66433c14b4db0cd1485df4ba85e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.408933 9580 raft_consensus.cc:2454] T 35970e99b91b48bda92e6f5173c31826 P 5acb9190154e4308bc0ec4293382e4dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dc2612096264b07a015817868d0435c in term 1.
I20240501 19:56:00.409162 9848 raft_consensus.cc:2454] T 35970e99b91b48bda92e6f5173c31826 P a786a66433c14b4db0cd1485df4ba85e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dc2612096264b07a015817868d0435c in term 1.
I20240501 19:56:00.409193 9650 leader_election.cc:302] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc2612096264b07a015817868d0435c, 5acb9190154e4308bc0ec4293382e4dd; no voters:
I20240501 19:56:00.409327 9919 raft_consensus.cc:2781] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:00.409536 9919 raft_consensus.cc:686] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c [term 1 LEADER]: Becoming Leader. State: Replica: 0dc2612096264b07a015817868d0435c, State: Running, Role: LEADER
I20240501 19:56:00.409821 9919 consensus_queue.cc:237] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acb9190154e4308bc0ec4293382e4dd" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44441 } } peers { permanent_uuid: "0dc2612096264b07a015817868d0435c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37723 } } peers { permanent_uuid: "a786a66433c14b4db0cd1485df4ba85e" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 38647 } }
I20240501 19:56:00.412707 9435 catalog_manager.cc:5484] T 35970e99b91b48bda92e6f5173c31826 P 0dc2612096264b07a015817868d0435c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dc2612096264b07a015817868d0435c (127.8.168.66). New cstate: current_term: 1 leader_uuid: "0dc2612096264b07a015817868d0435c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5acb9190154e4308bc0ec4293382e4dd" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dc2612096264b07a015817868d0435c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a786a66433c14b4db0cd1485df4ba85e" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 38647 } health_report { overall_health: UNKNOWN } } }
May 01 19:56:00 dist-test-slave-tzct krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.65@KRBTEST.COM
I20240501 19:56:00.451267 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.443659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.8.168.65:44441)
0501 19:56:00.443787 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:00.443791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:00.443949 (+ 158us) server_negotiation.cc:407] Connection header received
0501 19:56:00.444083 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:00.444085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:00.444131 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:00.444199 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:00.445006 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.445475 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.446076 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.446263 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.448291 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:00.448316 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:00.448330 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:00.448354 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:00.449860 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:00.450208 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:00.450214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:00.450215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:00.450279 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:00.450476 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:00.450479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:00.450481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:00.450653 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:00.450848 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:00.451000 (+ 152us) server_negotiation.cc:299] Negotiation successful
0501 19:56:00.451111 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
May 01 19:56:00 dist-test-slave-tzct krb5kdc[9390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593360, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 01 19:56:00 dist-test-slave-tzct krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593360, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.168.65@KRBTEST.COM
I20240501 19:56:00.474287 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.466786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.8.168.65:44441)
0501 19:56:00.466952 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:00.466956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:00.467042 (+ 86us) server_negotiation.cc:407] Connection header received
0501 19:56:00.467186 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:00.467191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:00.467245 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:00.467319 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:00.468102 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.468580 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.469273 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.469463 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.471548 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:00.471568 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:00.471576 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:00.471603 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:00.472962 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:00.473317 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:00.473321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:00.473322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:00.473365 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:00.473554 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:00.473557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:00.473559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:00.473688 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:00.473806 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:00.474022 (+ 216us) server_negotiation.cc:299] Negotiation successful
0501 19:56:00.474127 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
May 01 19:56:00 dist-test-slave-tzct krb5kdc[9390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 01 19:56:00 dist-test-slave-tzct krb5kdc[9390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.65@KRBTEST.COM
I20240501 19:56:00.496531 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:00.489525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.8.168.65:44441)
0501 19:56:00.489676 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:00.489680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:00.489771 (+ 91us) server_negotiation.cc:407] Connection header received
0501 19:56:00.489891 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:00.489895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:00.489944 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:00.490012 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:00.490784 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.491266 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:00.491918 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:00.492106 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:00.494043 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:00.494060 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:00.494062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:00.494087 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:00.495225 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:00.495565 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:00.495568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:00.495570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:00.495616 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:00.495810 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:00.495813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:00.495815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:00.495940 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:00.496051 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:00.496268 (+ 217us) server_negotiation.cc:299] Negotiation successful
0501 19:56:00.496370 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20240501 19:56:00.497856 9537 tablet_service.cc:2908] Rejecting scan request for tablet 35970e99b91b48bda92e6f5173c31826: Uninitialized: clean time has not yet been initialized
W20240501 19:56:00.498076 9895 tablet.cc:2369] T 35970e99b91b48bda92e6f5173c31826 P a786a66433c14b4db0cd1485df4ba85e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:00.499423 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 9497
I20240501 19:56:00.504329 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 9631
I20240501 19:56:00.508970 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 9765
I20240501 19:56:00.513167 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 9403
2024-05-01T19:56:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2132 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 19:56:00 dist-test-slave-tzct krb5kdc[9935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 19:56:00 dist-test-slave-tzct krb5kdc[9935](info): set up 2 sockets
May 01 19:56:00 dist-test-slave-tzct krb5kdc[9935](info): commencing operation
krb5kdc: starting...
W20240501 19:56:01.396749 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.863s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 19:56:01 dist-test-slave-tzct krb5kdc[9935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T19:56:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 19:56:01.510004 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:44223
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:42303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:44223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:01.618271 9948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:01.618502 9948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:01.618557 9948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:01.621626 9948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:01.621685 9948 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240501 19:56:01.621703 9948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:01.621722 9948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:01.621739 9948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:01.621759 9948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:01.626358 9948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:42303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:44223
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:44223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:01.627247 9948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:01.628074 9948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:01.635216 9956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:01.635596 9954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:01.636206 9953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:01.636773 9948 server_base.cc:1034] running on GCE node
I20240501 19:56:01.637274 9948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:01.638170 9948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:01.639322 9948 hybrid_clock.cc:648] HybridClock initialized: now 1714593361639309 us; error 32 us; skew 500 ppm
May 01 19:56:01 dist-test-slave-tzct krb5kdc[9935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593361, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:01.641999 9948 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 19:56:01.642922 9948 webserver.cc:460] Webserver started at http://127.8.168.126:40037/ using document root <none> and password file <none>
I20240501 19:56:01.643426 9948 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:01.643478 9948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:01.643627 9948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:01.645406 9948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "3532ee91f68948f88709952d2f63e643"
format_stamp: "Formatted at 2024-05-01 19:56:01 on dist-test-slave-tzct"
server_key: "15b81eb53eb7f82e9044c5e930143884"
server_key_iv: "c088fcf51d18dacc61b431a5ea8db54c"
server_key_version: "encryptionkey@0"
I20240501 19:56:01.645844 9948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "3532ee91f68948f88709952d2f63e643"
format_stamp: "Formatted at 2024-05-01 19:56:01 on dist-test-slave-tzct"
server_key: "15b81eb53eb7f82e9044c5e930143884"
server_key_iv: "c088fcf51d18dacc61b431a5ea8db54c"
server_key_version: "encryptionkey@0"
I20240501 19:56:01.649236 9948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:01.651495 9963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:01.652544 9948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:01.652659 9948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "3532ee91f68948f88709952d2f63e643"
format_stamp: "Formatted at 2024-05-01 19:56:01 on dist-test-slave-tzct"
server_key: "15b81eb53eb7f82e9044c5e930143884"
server_key_iv: "c088fcf51d18dacc61b431a5ea8db54c"
server_key_version: "encryptionkey@0"
I20240501 19:56:01.652803 9948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:01.667155 9948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:01.670121 9948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:01.670301 9948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:01.678460 9948 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:44223
I20240501 19:56:01.678467 10025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:44223 every 8 connection(s)
I20240501 19:56:01.679497 9948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:56:01.681989 10026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:01.685516 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 9948
I20240501 19:56:01.685606 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:56:01.685847 8865 external_mini_cluster.cc:1428] Setting key 3f92349f149dd204ba6eefc31a3e12ae
I20240501 19:56:01.687028 10026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643: Bootstrap starting.
I20240501 19:56:01.689111 10026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:01.689785 10026 log.cc:824] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:01.691449 10026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643: No bootstrap required, opened a new log
May 01 19:56:01 dist-test-slave-tzct krb5kdc[9935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:01.694676 10026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3532ee91f68948f88709952d2f63e643" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 44223 } }
I20240501 19:56:01.694909 10026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:01.694957 10026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3532ee91f68948f88709952d2f63e643, State: Initialized, Role: FOLLOWER
I20240501 19:56:01.695382 10026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3532ee91f68948f88709952d2f63e643" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 44223 } }
I20240501 19:56:01.695497 10026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:01.695550 10026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:01.695652 10026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:01.696399 10026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3532ee91f68948f88709952d2f63e643" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 44223 } }
I20240501 19:56:01.696702 10026 leader_election.cc:302] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3532ee91f68948f88709952d2f63e643; no voters:
I20240501 19:56:01.697008 10026 leader_election.cc:288] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:01.697091 10031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:01.697288 10031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [term 1 LEADER]: Becoming Leader. State: Replica: 3532ee91f68948f88709952d2f63e643, State: Running, Role: LEADER
I20240501 19:56:01.697600 10031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3532ee91f68948f88709952d2f63e643" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 44223 } }
I20240501 19:56:01.698179 10026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:01.698521 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:01.687081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.8.168.126:44223)
0501 19:56:01.687484 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:01.687494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:01.687530 (+ 36us) server_negotiation.cc:407] Connection header received
0501 19:56:01.688145 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:01.688165 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:01.688469 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:01.688834 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:01.689724 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:01.690543 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:01.691176 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:01.691464 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:01.693796 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:01.693816 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:01.693827 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:01.693863 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:01.695698 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:01.696086 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:01.696091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:01.696097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:01.696175 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:01.696373 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:01.696376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:01.696379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:01.696939 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:01.697198 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:01.697445 (+ 247us) server_negotiation.cc:299] Negotiation successful
0501 19:56:01.697882 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":135,"threads_started":1}
I20240501 19:56:01.698716 10031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3532ee91f68948f88709952d2f63e643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3532ee91f68948f88709952d2f63e643" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 44223 } } }
I20240501 19:56:01.698931 10031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:01.699173 10031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3532ee91f68948f88709952d2f63e643. Latest consensus state: current_term: 1 leader_uuid: "3532ee91f68948f88709952d2f63e643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3532ee91f68948f88709952d2f63e643" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 44223 } } }
I20240501 19:56:01.699301 10031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643 [sys.catalog]: This master's current role is: LEADER
W20240501 19:56:01.702929 10046 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:56:01.703001 10046 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:56:01.703302 10038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:01.704202 10038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:01.709223 10038 catalog_manager.cc:1347] Generated new cluster ID: 68b2f0edd63a47eb95d31fcd82543b75
I20240501 19:56:01.709379 10038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:01.765520 10038 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:01.766521 10038 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:01.785550 10038 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643: Generated new TSK 0
I20240501 19:56:01.786306 10038 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
I20240501 19:56:01.823513 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:44223
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:42303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:01.926369 10052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:01.926600 10052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:01.926677 10052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:01.929750 10052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:01.929812 10052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:01.929879 10052 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 19:56:01.929940 10052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:56:01.934115 10052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:42303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.168.126:44223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:01.934998 10052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:01.935817 10052 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:01.943709 10057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:01.943785 10060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:01.944324 10058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:01.944826 10052 server_base.cc:1034] running on GCE node
I20240501 19:56:01.945183 10052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:01.945709 10052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:01.946866 10052 hybrid_clock.cc:648] HybridClock initialized: now 1714593361946848 us; error 37 us; skew 500 ppm
May 01 19:56:01 dist-test-slave-tzct krb5kdc[9935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593361, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:01.949359 10052 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 19:56:01.950225 10052 webserver.cc:460] Webserver started at http://127.8.168.65:43629/ using document root <none> and password file <none>
I20240501 19:56:01.950717 10052 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:01.950779 10052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:01.950958 10052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:01.952688 10052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "2522521cb7d34173a740a39319669da1"
format_stamp: "Formatted at 2024-05-01 19:56:01 on dist-test-slave-tzct"
server_key: "12b869bf42585e77c8dcdd2dc925dc9c"
server_key_iv: "e81c313121dc68c9fddb26b606c697e8"
server_key_version: "encryptionkey@0"
I20240501 19:56:01.953186 10052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "2522521cb7d34173a740a39319669da1"
format_stamp: "Formatted at 2024-05-01 19:56:01 on dist-test-slave-tzct"
server_key: "12b869bf42585e77c8dcdd2dc925dc9c"
server_key_iv: "e81c313121dc68c9fddb26b606c697e8"
server_key_version: "encryptionkey@0"
I20240501 19:56:01.956411 10052 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:01.958633 10067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:01.959630 10052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:01.959734 10052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "2522521cb7d34173a740a39319669da1"
format_stamp: "Formatted at 2024-05-01 19:56:01 on dist-test-slave-tzct"
server_key: "12b869bf42585e77c8dcdd2dc925dc9c"
server_key_iv: "e81c313121dc68c9fddb26b606c697e8"
server_key_version: "encryptionkey@0"
I20240501 19:56:01.959846 10052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:01.977294 10052 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:01.980129 10052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:01.980293 10052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:01.981716 10052 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:01.981774 10052 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:01.981829 10052 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:01.981854 10052 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 19:56:01 dist-test-slave-tzct krb5kdc[9935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593361, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:01.991366 10052 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:37677
I20240501 19:56:01.991416 10182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:37677 every 8 connection(s)
I20240501 19:56:01.992399 10052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 19:56:01.994362 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:01.982563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:55413 (local address 127.8.168.126:44223)
0501 19:56:01.982682 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:01.982685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:01.983499 (+ 814us) server_negotiation.cc:407] Connection header received
0501 19:56:01.984407 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:01.984412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:01.984473 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:01.984532 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:01.986186 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:01.986891 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:01.987491 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:01.987685 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:01.990829 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:01.990855 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:01.990858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:01.990889 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:01.992487 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:01.993108 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:01.993111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:01.993113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:01.993171 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:01.993460 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:01.993463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:01.993464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:01.993621 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:01.993698 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:01.994122 (+ 424us) server_negotiation.cc:299] Negotiation successful
0501 19:56:01.994236 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240501 19:56:01.994920 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:01.982918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:44223 (local address 127.8.168.65:55413)
0501 19:56:01.983358 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:01.983394 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:01.984143 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:01.985007 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:01.985017 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:01.985427 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:01.986041 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:01.986055 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:01.987001 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:01.987005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:01.987382 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:01.987389 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:01.987600 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:01.988378 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:01.988425 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:01.990646 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:01.992631 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:01.992635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:01.992644 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:01.992978 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:01.993282 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:01.993285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:01.993286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:01.993366 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:01.993698 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:01.993702 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:01.993915 (+ 213us) client_negotiation.cc:769] Sending connection context
0501 19:56:01.994067 (+ 152us) client_negotiation.cc:240] Negotiation successful
0501 19:56:01.994298 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
I20240501 19:56:01.996188 10183 heartbeater.cc:347] Connected to a master server at 127.8.168.126:44223
I20240501 19:56:01.996325 10183 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:01.996858 10183 heartbeater.cc:511] Master 127.8.168.126:44223 requested a full tablet report, sending...
I20240501 19:56:01.998328 9979 ts_manager.cc:194] Registered new tserver with Master: 2522521cb7d34173a740a39319669da1 (127.8.168.65:37677)
I20240501 19:56:01.998818 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 10052
I20240501 19:56:01.998919 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 19:56:01.999169 8865 external_mini_cluster.cc:1428] Setting key 389243956872745de2f6f707e30ff6b6
I20240501 19:56:01.999761 9979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:55413
I20240501 19:56:02.004196 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:01.998339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:48049 (local address 127.8.168.126:44223)
0501 19:56:01.998458 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:01.998462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:01.998558 (+ 96us) server_negotiation.cc:407] Connection header received
0501 19:56:01.998668 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:01.998670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:01.998711 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:01.998817 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:01.999395 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:01.999961 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.000697 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.000901 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.001648 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:02.001663 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:02.001665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:02.001685 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:02.002939 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:02.003277 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:02.003281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:02.003283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:02.003323 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:02.003514 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:02.003517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:02.003519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:02.003644 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:02.003722 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:02.003958 (+ 236us) server_negotiation.cc:299] Negotiation successful
0501 19:56:02.004051 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20240501 19:56:02.004217 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:01.998259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:44223 (local address 127.8.168.65:48049)
0501 19:56:01.998486 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:01.998498 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:01.998576 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:01.998871 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:01.998874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:01.999041 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:01.999293 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:01.999301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.000058 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.000062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:02.000573 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.000584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.000714 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.001251 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:02.001274 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:02.001534 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:02.003037 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:02.003041 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:02.003043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:02.003196 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:02.003398 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:02.003401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:02.003403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:02.003443 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:02.003724 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:02.003727 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:02.003809 (+ 82us) client_negotiation.cc:769] Sending connection context
0501 19:56:02.003907 (+ 98us) client_negotiation.cc:240] Negotiation successful
0501 19:56:02.004030 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.8.168.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
I20240501 19:56:02.023546 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:44223
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:42303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:02.128850 10192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:02.129076 10192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:02.129149 10192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:02.132124 10192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:02.132184 10192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:02.132251 10192 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 19:56:02.132315 10192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:56:02.136360 10192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:42303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.168.126:44223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:02.137252 10192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:02.138098 10192 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:02.145663 10200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:02.146112 10198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:02.146723 10197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:02.147153 10192 server_base.cc:1034] running on GCE node
I20240501 19:56:02.147521 10192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:02.148053 10192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:02.149205 10192 hybrid_clock.cc:648] HybridClock initialized: now 1714593362149189 us; error 27 us; skew 500 ppm
May 01 19:56:02 dist-test-slave-tzct krb5kdc[9935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593362, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:02.151702 10192 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 19:56:02.152568 10192 webserver.cc:460] Webserver started at http://127.8.168.66:42795/ using document root <none> and password file <none>
I20240501 19:56:02.153131 10192 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:02.153185 10192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:02.153331 10192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:02.155023 10192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "2eeff5db83424480a26bea12c78d0e07"
format_stamp: "Formatted at 2024-05-01 19:56:02 on dist-test-slave-tzct"
server_key: "62fb7ce83845a2744b150944077ae227"
server_key_iv: "65c625929797918567304a3707d631cd"
server_key_version: "encryptionkey@0"
I20240501 19:56:02.155438 10192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "2eeff5db83424480a26bea12c78d0e07"
format_stamp: "Formatted at 2024-05-01 19:56:02 on dist-test-slave-tzct"
server_key: "62fb7ce83845a2744b150944077ae227"
server_key_iv: "65c625929797918567304a3707d631cd"
server_key_version: "encryptionkey@0"
I20240501 19:56:02.158675 10192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240501 19:56:02.160748 10207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:02.161720 10192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:02.161850 10192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "2eeff5db83424480a26bea12c78d0e07"
format_stamp: "Formatted at 2024-05-01 19:56:02 on dist-test-slave-tzct"
server_key: "62fb7ce83845a2744b150944077ae227"
server_key_iv: "65c625929797918567304a3707d631cd"
server_key_version: "encryptionkey@0"
I20240501 19:56:02.161935 10192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:02.176234 10192 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:02.178864 10192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:02.179018 10192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:02.180382 10192 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:02.180433 10192 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:02.180471 10192 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:02.180487 10192 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 19:56:02 dist-test-slave-tzct krb5kdc[9935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593362, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:02.189970 10192 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:44521
I20240501 19:56:02.190014 10322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:44521 every 8 connection(s)
I20240501 19:56:02.191058 10192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 19:56:02.192651 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.181015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:36517 (local address 127.8.168.126:44223)
0501 19:56:02.181154 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:02.181158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:02.181830 (+ 672us) server_negotiation.cc:407] Connection header received
0501 19:56:02.182645 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:02.182649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:02.182709 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:02.182809 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:02.184371 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.185058 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.185731 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.185907 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.189145 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:02.189167 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:02.189169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:02.189196 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:02.190833 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:02.191314 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:02.191318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:02.191320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:02.191375 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:02.191684 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:02.191687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:02.191689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:02.191885 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:02.191963 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:02.192417 (+ 454us) server_negotiation.cc:299] Negotiation successful
0501 19:56:02.192533 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240501 19:56:02.193215 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.181266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:44223 (local address 127.8.168.66:36517)
0501 19:56:02.181673 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:02.181716 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:02.182446 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:02.183042 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:02.183052 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:02.183550 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:02.184232 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.184249 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.185172 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.185176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:02.185614 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.185622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.185833 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.186497 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:02.186536 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:02.188986 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:02.190943 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:02.190948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:02.190956 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:02.191217 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:02.191472 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:02.191476 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:02.191477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:02.191576 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:02.191991 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:02.191995 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:02.192200 (+ 205us) client_negotiation.cc:769] Sending connection context
0501 19:56:02.192360 (+ 160us) client_negotiation.cc:240] Negotiation successful
0501 19:56:02.192592 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
I20240501 19:56:02.194356 10323 heartbeater.cc:347] Connected to a master server at 127.8.168.126:44223
I20240501 19:56:02.194492 10323 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:02.194955 10323 heartbeater.cc:511] Master 127.8.168.126:44223 requested a full tablet report, sending...
I20240501 19:56:02.195940 9980 ts_manager.cc:194] Registered new tserver with Master: 2eeff5db83424480a26bea12c78d0e07 (127.8.168.66:44521)
I20240501 19:56:02.196637 9980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:36517
I20240501 19:56:02.198470 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 10192
I20240501 19:56:02.198573 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 19:56:02.198859 8865 external_mini_cluster.cc:1428] Setting key 48d156c2126f885e613f236e2d50c80d
I20240501 19:56:02.203573 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.196252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:43797 (local address 127.8.168.126:44223)
0501 19:56:02.196841 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:02.196844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:02.196855 (+ 11us) server_negotiation.cc:407] Connection header received
0501 19:56:02.196881 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:02.196884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:02.196921 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:02.196998 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:02.198713 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.199321 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.200010 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.200178 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.200941 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:02.200957 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:02.200960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:02.200984 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:02.202296 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:02.202674 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:02.202677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:02.202678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:02.202713 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:02.202896 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:02.202898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:02.202899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:02.203039 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:02.203108 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:02.203375 (+ 267us) server_negotiation.cc:299] Negotiation successful
0501 19:56:02.203462 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520}
I20240501 19:56:02.203629 10329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.196159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:44223 (local address 127.8.168.66:43797)
0501 19:56:02.196410 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:02.196423 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:02.196520 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:02.197014 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:02.197016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:02.198316 (+ 1300us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:02.198612 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.198622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.199410 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.199413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:02.199911 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.199921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.200023 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.200531 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:02.200547 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:02.200837 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:02.202395 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:02.202399 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:02.202401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:02.202586 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:02.202777 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:02.202780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:02.202782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:02.202829 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:02.203113 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:02.203117 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:02.203196 (+ 79us) client_negotiation.cc:769] Sending connection context
0501 19:56:02.203317 (+ 121us) client_negotiation.cc:240] Negotiation successful
0501 19:56:02.203440 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.8.168.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
I20240501 19:56:02.223387 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:44223
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:42303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:02.331140 10332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:02.331358 10332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:02.331439 10332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:02.334975 10332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:02.335049 10332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:02.335121 10332 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 19:56:02.335184 10332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:56:02.339340 10332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:42303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.168.126:44223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:02.340206 10332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:02.341123 10332 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:02.348948 10337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:02.349406 10338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:02.349803 10340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:02.350464 10332 server_base.cc:1034] running on GCE node
I20240501 19:56:02.350867 10332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:02.351414 10332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:02.352591 10332 hybrid_clock.cc:648] HybridClock initialized: now 1714593362352573 us; error 30 us; skew 500 ppm
May 01 19:56:02 dist-test-slave-tzct krb5kdc[9935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593362, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:02.355255 10332 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 19:56:02.356225 10332 webserver.cc:460] Webserver started at http://127.8.168.67:41083/ using document root <none> and password file <none>
I20240501 19:56:02.356824 10332 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:02.356895 10332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:02.357120 10332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:02.358927 10332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "121318f4216144fa8f268b1c0cf91522"
format_stamp: "Formatted at 2024-05-01 19:56:02 on dist-test-slave-tzct"
server_key: "df587ceb86e457a742dc04dc471c32df"
server_key_iv: "c7e93392970b78c2f3e9213fcd44a9cf"
server_key_version: "encryptionkey@0"
I20240501 19:56:02.359411 10332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "121318f4216144fa8f268b1c0cf91522"
format_stamp: "Formatted at 2024-05-01 19:56:02 on dist-test-slave-tzct"
server_key: "df587ceb86e457a742dc04dc471c32df"
server_key_iv: "c7e93392970b78c2f3e9213fcd44a9cf"
server_key_version: "encryptionkey@0"
I20240501 19:56:02.362799 10332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:02.365082 10347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:02.366133 10332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:02.366233 10332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "121318f4216144fa8f268b1c0cf91522"
format_stamp: "Formatted at 2024-05-01 19:56:02 on dist-test-slave-tzct"
server_key: "df587ceb86e457a742dc04dc471c32df"
server_key_iv: "c7e93392970b78c2f3e9213fcd44a9cf"
server_key_version: "encryptionkey@0"
I20240501 19:56:02.366309 10332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:02.381058 10332 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:02.384075 10332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:02.384240 10332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:02.385668 10332 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:02.385726 10332 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:02.385780 10332 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:02.385809 10332 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 19:56:02 dist-test-slave-tzct krb5kdc[9935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593362, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:02.396332 10332 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:34959
I20240501 19:56:02.396456 10462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:34959 every 8 connection(s)
I20240501 19:56:02.397557 10332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 19:56:02.397958 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.386244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:44413 (local address 127.8.168.126:44223)
0501 19:56:02.386397 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:02.386402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:02.387071 (+ 669us) server_negotiation.cc:407] Connection header received
0501 19:56:02.387878 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:02.387882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:02.387940 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:02.388036 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:02.389889 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.390571 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.391218 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.391400 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.393758 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:02.393813 (+ 55us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:02.393817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:02.393852 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:02.395724 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:02.396353 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:02.396357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:02.396359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:02.396450 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:02.396836 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:02.396838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:02.396840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:02.397001 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:02.397096 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:02.397711 (+ 615us) server_negotiation.cc:299] Negotiation successful
0501 19:56:02.397828 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20240501 19:56:02.398720 10356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.386516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:44223 (local address 127.8.168.67:44413)
0501 19:56:02.386926 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:02.386967 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:02.387670 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:02.388351 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:02.388365 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:02.388915 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:02.389735 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.389755 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.390690 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.390694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:02.391106 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.391114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.391328 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.392025 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:02.392053 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:02.393585 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:02.395871 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:02.395877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:02.395909 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:02.396234 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:02.396584 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:02.396588 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:02.396590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:02.396703 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:02.397101 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:02.397107 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:02.397401 (+ 294us) client_negotiation.cc:769] Sending connection context
0501 19:56:02.397640 (+ 239us) client_negotiation.cc:240] Negotiation successful
0501 19:56:02.397948 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":96,"threads_started":1}
I20240501 19:56:02.399873 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 10332
I20240501 19:56:02.399938 10463 heartbeater.cc:347] Connected to a master server at 127.8.168.126:44223
I20240501 19:56:02.399982 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 19:56:02.400059 10463 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:02.400238 8865 external_mini_cluster.cc:1428] Setting key f57256c1acce7d8d68f62ef66d3618f5
I20240501 19:56:02.400580 10463 heartbeater.cc:511] Master 127.8.168.126:44223 requested a full tablet report, sending...
I20240501 19:56:02.401753 9980 ts_manager.cc:194] Registered new tserver with Master: 121318f4216144fa8f268b1c0cf91522 (127.8.168.67:34959)
I20240501 19:56:02.402428 9980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:44413
I20240501 19:56:02.402518 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 19:56:02.408892 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.402185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:60687 (local address 127.8.168.126:44223)
0501 19:56:02.402307 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:02.402312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:02.402544 (+ 232us) server_negotiation.cc:407] Connection header received
0501 19:56:02.402861 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:02.402865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:02.402916 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:02.402971 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:02.403731 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.404326 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.404936 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.405106 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.405942 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:02.405957 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:02.405959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:02.405979 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:02.407501 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:02.407872 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:02.407875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:02.407877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:02.407923 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:02.408211 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:02.408213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:02.408214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:02.408342 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:02.408408 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:02.408652 (+ 244us) server_negotiation.cc:299] Negotiation successful
0501 19:56:02.408781 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240501 19:56:02.408907 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.402106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:44223 (local address 127.8.168.67:60687)
0501 19:56:02.402466 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:02.402481 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:02.402614 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:02.403115 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:02.403118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:02.403304 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:02.403593 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.403602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.404430 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.404433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:02.404839 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.404847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.404944 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.405559 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:02.405570 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:02.405833 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:02.407624 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:02.407627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:02.407629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:02.407786 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:02.408043 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:02.408047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:02.408049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:02.408114 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:02.408415 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:02.408418 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:02.408511 (+ 93us) client_negotiation.cc:769] Sending connection context
0501 19:56:02.408610 (+ 99us) client_negotiation.cc:240] Negotiation successful
0501 19:56:02.408756 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":93,"threads_started":1}
May 01 19:56:02 dist-test-slave-tzct krb5kdc[9935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593362, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 01 19:56:02 dist-test-slave-tzct krb5kdc[9935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593362, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:02.433614 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.426097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.8.168.126:44223)
0501 19:56:02.426255 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:02.426260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:02.426275 (+ 15us) server_negotiation.cc:407] Connection header received
0501 19:56:02.426325 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:02.426328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:02.426383 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:02.426481 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:02.427505 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.428246 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.428936 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.429118 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.431104 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:02.431124 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:02.431126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:02.431150 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:02.432270 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:02.432631 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:02.432634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:02.432636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:02.432680 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:02.432899 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:02.432902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:02.432904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:02.433043 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:02.433158 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:02.433354 (+ 196us) server_negotiation.cc:299] Negotiation successful
0501 19:56:02.433457 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240501 19:56:02.436547 9980 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:56:02.439126 9980 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:02.451452 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.447410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:44521 (local address 127.0.0.1:51606)
0501 19:56:02.447632 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:02.447644 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:02.447725 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:02.448525 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:02.448528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:02.448547 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:02.448889 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.448900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.450180 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.450184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:02.451045 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.451056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.451136 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.451150 (+ 14us) client_negotiation.cc:769] Sending connection context
0501 19:56:02.451221 (+ 71us) client_negotiation.cc:240] Negotiation successful
0501 19:56:02.451308 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20240501 19:56:02.451766 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.446875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58320 (local address 127.8.168.65:37677)
0501 19:56:02.447207 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:02.447213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:02.447247 (+ 34us) server_negotiation.cc:407] Connection header received
0501 19:56:02.447368 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:02.447373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:02.447564 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:02.447720 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:02.448375 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.449509 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.450576 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.451118 (+ 542us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.451239 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:02.451543 (+ 304us) server_negotiation.cc:299] Negotiation successful
0501 19:56:02.451637 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":117,"threads_started":1}
I20240501 19:56:02.452019 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.446749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:37677 (local address 127.0.0.1:58320)
0501 19:56:02.447074 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:02.447115 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:02.447220 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:02.447933 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:02.447939 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:02.447962 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:02.448253 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.448262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.449610 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.449613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:02.450423 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.450434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.450560 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.450626 (+ 66us) client_negotiation.cc:769] Sending connection context
0501 19:56:02.451356 (+ 730us) client_negotiation.cc:240] Negotiation successful
0501 19:56:02.451419 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":85,"threads_started":1}
I20240501 19:56:02.452270 10484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.447729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51606 (local address 127.8.168.66:44521)
0501 19:56:02.448054 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:02.448059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:02.448081 (+ 22us) server_negotiation.cc:407] Connection header received
0501 19:56:02.448162 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:02.448167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:02.448328 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:02.448464 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:02.449153 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.450063 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.451379 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.451885 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.451993 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:02.452059 (+ 66us) server_negotiation.cc:299] Negotiation successful
0501 19:56:02.452142 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":121,"threads_started":1}
I20240501 19:56:02.452875 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.447486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:34959 (local address 127.0.0.1:44062)
0501 19:56:02.449556 (+ 2070us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:02.449569 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:02.449658 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:02.450075 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:02.450077 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:02.450096 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:02.450335 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.450342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.451535 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.451539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:02.452507 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.452517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.452614 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.452630 (+ 16us) client_negotiation.cc:769] Sending connection context
0501 19:56:02.452674 (+ 44us) client_negotiation.cc:240] Negotiation successful
0501 19:56:02.452766 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2003,"thread_start_us":97,"threads_started":1}
I20240501 19:56:02.454361 10257 tablet_service.cc:1467] Processing CreateTablet for tablet 8e47e1bf126a45b8bcd58474f7c39b20 (DEFAULT_TABLE table=test-table [id=125c670091ad4eadaa587cc77dc68a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.454355 10117 tablet_service.cc:1467] Processing CreateTablet for tablet 8e47e1bf126a45b8bcd58474f7c39b20 (DEFAULT_TABLE table=test-table [id=125c670091ad4eadaa587cc77dc68a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.455191 10257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e47e1bf126a45b8bcd58474f7c39b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.455191 10117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e47e1bf126a45b8bcd58474f7c39b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.460278 10487 tablet_bootstrap.cc:492] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07: Bootstrap starting.
I20240501 19:56:02.460307 10486 tablet_bootstrap.cc:492] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1: Bootstrap starting.
I20240501 19:56:02.462275 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.447876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.8.168.67:34959)
0501 19:56:02.448210 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:02.448215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:02.449642 (+ 1427us) server_negotiation.cc:407] Connection header received
0501 19:56:02.449789 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:02.449794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:02.449947 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:02.450086 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:02.450435 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.451422 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.452635 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.461830 (+ 9195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.461956 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:02.462036 (+ 80us) server_negotiation.cc:299] Negotiation successful
0501 19:56:02.462134 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":93,"threads_started":1}
I20240501 19:56:02.462839 10487 tablet_bootstrap.cc:654] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.463168 10486 tablet_bootstrap.cc:654] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.463701 10487 log.cc:824] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:02.463900 10486 log.cc:824] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:02.464896 10397 tablet_service.cc:1467] Processing CreateTablet for tablet 8e47e1bf126a45b8bcd58474f7c39b20 (DEFAULT_TABLE table=test-table [id=125c670091ad4eadaa587cc77dc68a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.465636 10486 tablet_bootstrap.cc:492] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1: No bootstrap required, opened a new log
I20240501 19:56:02.465792 10487 tablet_bootstrap.cc:492] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07: No bootstrap required, opened a new log
I20240501 19:56:02.465811 10486 ts_tablet_manager.cc:1367] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 19:56:02.465950 10397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e47e1bf126a45b8bcd58474f7c39b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.465965 10487 ts_tablet_manager.cc:1367] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 19:56:02.468206 10486 raft_consensus.cc:348] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.468390 10486 raft_consensus.cc:374] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.468468 10486 raft_consensus.cc:729] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2522521cb7d34173a740a39319669da1, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.468930 10486 consensus_queue.cc:260] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.469122 10487 raft_consensus.cc:348] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.469553 10487 raft_consensus.cc:374] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.469660 10487 raft_consensus.cc:729] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eeff5db83424480a26bea12c78d0e07, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.469897 10486 ts_tablet_manager.cc:1398] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:56:02.470360 10183 heartbeater.cc:503] Master 127.8.168.126:44223 was elected leader, sending a full tablet report...
I20240501 19:56:02.470292 10487 consensus_queue.cc:260] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.470944 10323 heartbeater.cc:503] Master 127.8.168.126:44223 was elected leader, sending a full tablet report...
I20240501 19:56:02.471103 10487 ts_tablet_manager.cc:1398] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 19:56:02.475116 10492 tablet_bootstrap.cc:492] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522: Bootstrap starting.
I20240501 19:56:02.477466 10492 tablet_bootstrap.cc:654] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.478116 10492 log.cc:824] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:02.479764 10492 tablet_bootstrap.cc:492] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522: No bootstrap required, opened a new log
I20240501 19:56:02.479926 10492 ts_tablet_manager.cc:1367] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:02.482652 10492 raft_consensus.cc:348] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.482832 10492 raft_consensus.cc:374] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.482880 10492 raft_consensus.cc:729] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121318f4216144fa8f268b1c0cf91522, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.483227 10492 consensus_queue.cc:260] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.483865 10463 heartbeater.cc:503] Master 127.8.168.126:44223 was elected leader, sending a full tablet report...
I20240501 19:56:02.484000 10492 ts_tablet_manager.cc:1398] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
W20240501 19:56:02.493544 10184 tablet.cc:2369] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:02.535199 10490 raft_consensus.cc:482] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:02.535403 10490 raft_consensus.cc:504] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.536520 10490 leader_election.cc:288] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2eeff5db83424480a26bea12c78d0e07 (127.8.168.66:44521), 121318f4216144fa8f268b1c0cf91522 (127.8.168.67:34959)
I20240501 19:56:02.540409 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.536907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:34959 (local address 127.8.168.65:55019)
0501 19:56:02.537201 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:02.537212 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:02.537333 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:02.537825 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:02.537828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:02.537843 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:02.538069 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.538076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.539028 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.539031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:02.539931 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.539942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.540064 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.540086 (+ 22us) client_negotiation.cc:769] Sending connection context
0501 19:56:02.540136 (+ 50us) client_negotiation.cc:240] Negotiation successful
0501 19:56:02.540196 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"spinlock_wait_cycles":9856,"thread_start_us":90,"threads_started":1}
I20240501 19:56:02.540408 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.536786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:44521 (local address 127.8.168.65:47171)
0501 19:56:02.537180 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:02.537194 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:02.537333 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:02.537627 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:02.537630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:02.537650 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:02.537882 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.537890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.538950 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.538952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:02.539931 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.539942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.540066 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.540086 (+ 20us) client_negotiation.cc:769] Sending connection context
0501 19:56:02.540136 (+ 50us) client_negotiation.cc:240] Negotiation successful
0501 19:56:02.540196 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"spinlock_wait_cycles":14336,"thread_start_us":86,"threads_started":1}
I20240501 19:56:02.540853 10484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.536959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:47171 (local address 127.8.168.66:44521)
0501 19:56:02.537095 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:02.537100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:02.537286 (+ 186us) server_negotiation.cc:407] Connection header received
0501 19:56:02.537440 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:02.537443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:02.537501 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:02.537563 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:02.538014 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.538833 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.540063 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.540571 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.540602 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:02.540655 (+ 53us) server_negotiation.cc:299] Negotiation successful
0501 19:56:02.540711 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240501 19:56:02.540882 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.537009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:55019 (local address 127.8.168.67:34959)
0501 19:56:02.537140 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:02.537144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:02.537316 (+ 172us) server_negotiation.cc:407] Connection header received
0501 19:56:02.537668 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:02.537671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:02.537715 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:02.537770 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:02.538190 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.538932 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.540063 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.540564 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.540602 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:02.540654 (+ 52us) server_negotiation.cc:299] Negotiation successful
0501 19:56:02.540785 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240501 19:56:02.541450 10273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e47e1bf126a45b8bcd58474f7c39b20" candidate_uuid: "2522521cb7d34173a740a39319669da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eeff5db83424480a26bea12c78d0e07" is_pre_election: true
I20240501 19:56:02.541452 10416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e47e1bf126a45b8bcd58474f7c39b20" candidate_uuid: "2522521cb7d34173a740a39319669da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121318f4216144fa8f268b1c0cf91522" is_pre_election: true
I20240501 19:56:02.541740 10273 raft_consensus.cc:2454] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2522521cb7d34173a740a39319669da1 in term 0.
I20240501 19:56:02.541793 10416 raft_consensus.cc:2454] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2522521cb7d34173a740a39319669da1 in term 0.
I20240501 19:56:02.542213 10068 leader_election.cc:302] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2522521cb7d34173a740a39319669da1, 2eeff5db83424480a26bea12c78d0e07; no voters:
I20240501 19:56:02.542472 10490 raft_consensus.cc:2781] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:02.542542 10490 raft_consensus.cc:482] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:02.542577 10490 raft_consensus.cc:3037] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:02.583916 10490 raft_consensus.cc:504] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.584534 10490 leader_election.cc:288] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [CANDIDATE]: Term 1 election: Requested vote from peers 2eeff5db83424480a26bea12c78d0e07 (127.8.168.66:44521), 121318f4216144fa8f268b1c0cf91522 (127.8.168.67:34959)
I20240501 19:56:02.585103 10416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e47e1bf126a45b8bcd58474f7c39b20" candidate_uuid: "2522521cb7d34173a740a39319669da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121318f4216144fa8f268b1c0cf91522"
I20240501 19:56:02.585115 10273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e47e1bf126a45b8bcd58474f7c39b20" candidate_uuid: "2522521cb7d34173a740a39319669da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eeff5db83424480a26bea12c78d0e07"
I20240501 19:56:02.585251 10273 raft_consensus.cc:3037] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:02.585242 10416 raft_consensus.cc:3037] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:02.586239 10416 raft_consensus.cc:2454] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2522521cb7d34173a740a39319669da1 in term 1.
I20240501 19:56:02.586465 10273 raft_consensus.cc:2454] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2522521cb7d34173a740a39319669da1 in term 1.
I20240501 19:56:02.586585 10069 leader_election.cc:302] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121318f4216144fa8f268b1c0cf91522, 2522521cb7d34173a740a39319669da1; no voters:
I20240501 19:56:02.586808 10490 raft_consensus.cc:2781] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:02.586983 10490 raft_consensus.cc:686] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [term 1 LEADER]: Becoming Leader. State: Replica: 2522521cb7d34173a740a39319669da1, State: Running, Role: LEADER
I20240501 19:56:02.587359 10490 consensus_queue.cc:237] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.590394 9980 catalog_manager.cc:5484] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2522521cb7d34173a740a39319669da1 (127.8.168.65). New cstate: current_term: 1 leader_uuid: "2522521cb7d34173a740a39319669da1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:02.608150 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.605345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58334 (local address 127.8.168.65:37677)
0501 19:56:02.605473 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:02.605476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:02.605490 (+ 14us) server_negotiation.cc:407] Connection header received
0501 19:56:02.605652 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:02.605654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:02.605707 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:02.605801 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:02.606123 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.606611 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.607216 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.607349 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.607398 (+ 49us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:02.607810 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:02.607892 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:02.607970 (+ 78us) server_negotiation.cc:299] Negotiation successful
0501 19:56:02.608014 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20240501 19:56:02.613420 10416 raft_consensus.cc:1261] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522 [term 1 FOLLOWER]: Refusing update from remote peer 2522521cb7d34173a740a39319669da1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:02.613420 10273 raft_consensus.cc:1261] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07 [term 1 FOLLOWER]: Refusing update from remote peer 2522521cb7d34173a740a39319669da1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:02.614053 10490 consensus_queue.cc:1035] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:02.614239 10497 consensus_queue.cc:1035] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:02.623134 10502 mvcc.cc:205] Tried to move back new op lower bound from 7022974413258100736 to 7022974413160628224. Current Snapshot: MvccSnapshot[applied={T|T < 7022974413258100736 or (T in {7022974413258100736})}]
I20240501 19:56:02.630098 9979 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44196:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:02.630251 9979 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44196:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:02.632617 9979 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643: Sending DeleteTablet for 3 replicas of tablet 8e47e1bf126a45b8bcd58474f7c39b20
I20240501 19:56:02.633502 10397 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e47e1bf126a45b8bcd58474f7c39b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:02 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:44062
I20240501 19:56:02.633710 10257 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e47e1bf126a45b8bcd58474f7c39b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:02 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:51606
I20240501 19:56:02.633991 10117 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e47e1bf126a45b8bcd58474f7c39b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:02 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:58320
I20240501 19:56:02.634450 10512 tablet_replica.cc:330] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1: stopping tablet replica
I20240501 19:56:02.634373 9979 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:56:02.634719 10510 tablet_replica.cc:330] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522: stopping tablet replica
W20240501 19:56:02.634801 9979 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:02.634852 10512 raft_consensus.cc:2229] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:02.635017 10510 raft_consensus.cc:2229] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:02.635252 10512 raft_consensus.cc:2258] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:02.635344 10510 raft_consensus.cc:2258] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:02.636245 10511 tablet_replica.cc:330] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07: stopping tablet replica
I20240501 19:56:02.636479 10512 ts_tablet_manager.cc:1875] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:02.636540 10511 raft_consensus.cc:2229] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:02.636926 10511 raft_consensus.cc:2258] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:02.639011 10510 ts_tablet_manager.cc:1875] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:02.639011 10511 ts_tablet_manager.cc:1875] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:02.641389 10117 tablet_service.cc:1467] Processing CreateTablet for tablet a9052c6dab394e03b88ee44058a88cae (DEFAULT_TABLE table=test-table [id=c5c7214f9f77445a989ef7a4892f7fa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.641642 10397 tablet_service.cc:1467] Processing CreateTablet for tablet a9052c6dab394e03b88ee44058a88cae (DEFAULT_TABLE table=test-table [id=c5c7214f9f77445a989ef7a4892f7fa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.641705 10257 tablet_service.cc:1467] Processing CreateTablet for tablet a9052c6dab394e03b88ee44058a88cae (DEFAULT_TABLE table=test-table [id=c5c7214f9f77445a989ef7a4892f7fa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:02.680167 10117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9052c6dab394e03b88ee44058a88cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.682575 10486 tablet_bootstrap.cc:492] T a9052c6dab394e03b88ee44058a88cae P 2522521cb7d34173a740a39319669da1: Bootstrap starting.
I20240501 19:56:02.682984 10257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9052c6dab394e03b88ee44058a88cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.683549 10486 tablet_bootstrap.cc:654] T a9052c6dab394e03b88ee44058a88cae P 2522521cb7d34173a740a39319669da1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.685230 10487 tablet_bootstrap.cc:492] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07: Bootstrap starting.
I20240501 19:56:02.686146 10487 tablet_bootstrap.cc:654] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.686174 10397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9052c6dab394e03b88ee44058a88cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.687855 10486 tablet_bootstrap.cc:492] T a9052c6dab394e03b88ee44058a88cae P 2522521cb7d34173a740a39319669da1: No bootstrap required, opened a new log
I20240501 19:56:02.687955 10486 ts_tablet_manager.cc:1367] T a9052c6dab394e03b88ee44058a88cae P 2522521cb7d34173a740a39319669da1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:02.688033 10487 tablet_bootstrap.cc:492] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07: No bootstrap required, opened a new log
I20240501 19:56:02.688115 10487 ts_tablet_manager.cc:1367] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:02.688426 10486 raft_consensus.cc:348] T a9052c6dab394e03b88ee44058a88cae P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.688529 10486 raft_consensus.cc:374] T a9052c6dab394e03b88ee44058a88cae P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.688491 10487 raft_consensus.cc:348] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.688557 10486 raft_consensus.cc:729] T a9052c6dab394e03b88ee44058a88cae P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2522521cb7d34173a740a39319669da1, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.688575 10487 raft_consensus.cc:374] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.688602 10487 raft_consensus.cc:729] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eeff5db83424480a26bea12c78d0e07, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.688710 10487 consensus_queue.cc:260] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.688683 10486 consensus_queue.cc:260] T a9052c6dab394e03b88ee44058a88cae P 2522521cb7d34173a740a39319669da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.688978 10486 ts_tablet_manager.cc:1398] T a9052c6dab394e03b88ee44058a88cae P 2522521cb7d34173a740a39319669da1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:02.688978 10487 ts_tablet_manager.cc:1398] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240501 19:56:02.692206 10324 tablet.cc:2369] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:02.724363 10491 raft_consensus.cc:482] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:02.724493 10491 raft_consensus.cc:504] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.725690 10491 leader_election.cc:288] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2522521cb7d34173a740a39319669da1 (127.8.168.65:37677), 121318f4216144fa8f268b1c0cf91522 (127.8.168.67:34959)
I20240501 19:56:02.729041 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.725886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:37677 (local address 127.8.168.66:46597)
0501 19:56:02.726184 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:02.726197 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:02.726353 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:02.726698 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:02.726701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:02.726715 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:02.726932 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.726940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.727954 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.727957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:02.728555 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.728565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.728678 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.728706 (+ 28us) client_negotiation.cc:769] Sending connection context
0501 19:56:02.728806 (+ 100us) client_negotiation.cc:240] Negotiation successful
0501 19:56:02.728878 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":82,"threads_started":1}
I20240501 19:56:02.729458 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.726054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:46597 (local address 127.8.168.65:37677)
0501 19:56:02.726180 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:02.726184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:02.726271 (+ 87us) server_negotiation.cc:407] Connection header received
0501 19:56:02.726478 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:02.726482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:02.726527 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:02.726565 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:02.727060 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.727823 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.728689 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.729190 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.729231 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:02.729284 (+ 53us) server_negotiation.cc:299] Negotiation successful
0501 19:56:02.729340 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20240501 19:56:02.729899 10137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9052c6dab394e03b88ee44058a88cae" candidate_uuid: "2eeff5db83424480a26bea12c78d0e07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2522521cb7d34173a740a39319669da1" is_pre_election: true
I20240501 19:56:02.730134 10137 raft_consensus.cc:2454] T a9052c6dab394e03b88ee44058a88cae P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eeff5db83424480a26bea12c78d0e07 in term 0.
I20240501 19:56:02.730237 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.725934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:34959 (local address 127.8.168.66:49157)
0501 19:56:02.726316 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:02.726326 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:02.726389 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:02.726625 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:02.726628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:02.726646 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:02.726932 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.726942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.728913 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.728917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:02.729838 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.729849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.729962 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.729979 (+ 17us) client_negotiation.cc:769] Sending connection context
0501 19:56:02.730023 (+ 44us) client_negotiation.cc:240] Negotiation successful
0501 19:56:02.730080 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":59,"threads_started":1}
I20240501 19:56:02.730665 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.726267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:49157 (local address 127.8.168.67:34959)
0501 19:56:02.726386 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:02.726391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:02.726406 (+ 15us) server_negotiation.cc:407] Connection header received
0501 19:56:02.726460 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:02.726463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:02.726515 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:02.726597 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:02.727991 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.728780 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.729975 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.730399 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.730439 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:02.730496 (+ 57us) server_negotiation.cc:299] Negotiation successful
0501 19:56:02.730562 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240501 19:56:02.730615 10208 leader_election.cc:302] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2522521cb7d34173a740a39319669da1, 2eeff5db83424480a26bea12c78d0e07; no voters:
I20240501 19:56:02.730878 10491 raft_consensus.cc:2781] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:02.730947 10491 raft_consensus.cc:482] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:02.730984 10491 raft_consensus.cc:3037] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Advancing to term 1
W20240501 19:56:02.731444 10209 leader_election.cc:341] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 121318f4216144fa8f268b1c0cf91522 (127.8.168.67:34959): Not found: Tablet not found: a9052c6dab394e03b88ee44058a88cae
I20240501 19:56:02.782807 10510 ts_tablet_manager.cc:1888] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:02.782799 10512 ts_tablet_manager.cc:1888] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:02.782930 10510 log.cc:1196] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/8e47e1bf126a45b8bcd58474f7c39b20
I20240501 19:56:02.782929 10512 log.cc:1196] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-0/wal/wals/8e47e1bf126a45b8bcd58474f7c39b20
I20240501 19:56:02.783248 10510 ts_tablet_manager.cc:1909] T 8e47e1bf126a45b8bcd58474f7c39b20 P 121318f4216144fa8f268b1c0cf91522: Deleting consensus metadata
I20240501 19:56:02.783289 10512 ts_tablet_manager.cc:1909] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2522521cb7d34173a740a39319669da1: Deleting consensus metadata
I20240501 19:56:02.783697 10511 ts_tablet_manager.cc:1888] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:02.783797 10511 log.cc:1196] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/8e47e1bf126a45b8bcd58474f7c39b20
I20240501 19:56:02.784117 10511 ts_tablet_manager.cc:1909] T 8e47e1bf126a45b8bcd58474f7c39b20 P 2eeff5db83424480a26bea12c78d0e07: Deleting consensus metadata
I20240501 19:56:02.784327 10491 raft_consensus.cc:504] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.784773 10492 tablet_bootstrap.cc:492] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522: Bootstrap starting.
I20240501 19:56:02.785374 10491 leader_election.cc:288] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [CANDIDATE]: Term 1 election: Requested vote from peers 2522521cb7d34173a740a39319669da1 (127.8.168.65:37677), 121318f4216144fa8f268b1c0cf91522 (127.8.168.67:34959)
I20240501 19:56:02.785708 10492 tablet_bootstrap.cc:654] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.786029 10137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9052c6dab394e03b88ee44058a88cae" candidate_uuid: "2eeff5db83424480a26bea12c78d0e07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2522521cb7d34173a740a39319669da1"
I20240501 19:56:02.786144 10137 raft_consensus.cc:3037] T a9052c6dab394e03b88ee44058a88cae P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:02.786484 10416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9052c6dab394e03b88ee44058a88cae" candidate_uuid: "2eeff5db83424480a26bea12c78d0e07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121318f4216144fa8f268b1c0cf91522"
W20240501 19:56:02.787086 10209 leader_election.cc:341] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 121318f4216144fa8f268b1c0cf91522 (127.8.168.67:34959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:02.787397 10492 tablet_bootstrap.cc:492] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522: No bootstrap required, opened a new log
I20240501 19:56:02.787468 10492 ts_tablet_manager.cc:1367] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:02.787861 10492 raft_consensus.cc:348] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.787961 10492 raft_consensus.cc:374] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.787989 10492 raft_consensus.cc:729] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121318f4216144fa8f268b1c0cf91522, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.788101 10492 consensus_queue.cc:260] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.788345 10492 ts_tablet_manager.cc:1398] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:02.788900 9965 catalog_manager.cc:4830] TS 121318f4216144fa8f268b1c0cf91522 (127.8.168.67:34959): tablet 8e47e1bf126a45b8bcd58474f7c39b20 (table test-table [id=125c670091ad4eadaa587cc77dc68a5b]) successfully deleted
I20240501 19:56:02.788923 9964 catalog_manager.cc:4830] TS 2522521cb7d34173a740a39319669da1 (127.8.168.65:37677): tablet 8e47e1bf126a45b8bcd58474f7c39b20 (table test-table [id=125c670091ad4eadaa587cc77dc68a5b]) successfully deleted
I20240501 19:56:02.801327 9964 catalog_manager.cc:4830] TS 2eeff5db83424480a26bea12c78d0e07 (127.8.168.66:44521): tablet 8e47e1bf126a45b8bcd58474f7c39b20 (table test-table [id=125c670091ad4eadaa587cc77dc68a5b]) successfully deleted
I20240501 19:56:02.880069 10137 raft_consensus.cc:2454] T a9052c6dab394e03b88ee44058a88cae P 2522521cb7d34173a740a39319669da1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eeff5db83424480a26bea12c78d0e07 in term 1.
I20240501 19:56:02.880509 10208 leader_election.cc:302] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2522521cb7d34173a740a39319669da1, 2eeff5db83424480a26bea12c78d0e07; no voters: 121318f4216144fa8f268b1c0cf91522
I20240501 19:56:02.880750 10491 raft_consensus.cc:2781] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:02.880970 10491 raft_consensus.cc:686] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [term 1 LEADER]: Becoming Leader. State: Replica: 2eeff5db83424480a26bea12c78d0e07, State: Running, Role: LEADER
I20240501 19:56:02.881242 10491 consensus_queue.cc:237] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.883812 9980 catalog_manager.cc:5484] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2eeff5db83424480a26bea12c78d0e07 (127.8.168.66). New cstate: current_term: 1 leader_uuid: "2eeff5db83424480a26bea12c78d0e07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:02.898787 10464 tablet.cc:2369] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522: Can't schedule compaction. Clean time has not been advanced past its initial value.
May 01 19:56:02 dist-test-slave-tzct krb5kdc[9935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593361, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:02.970126 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.958419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:44223 (local address 127.0.0.1:44210)
0501 19:56:02.958693 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:02.958707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:02.958824 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:02.959061 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:02.959064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:02.959579 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:02.959794 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.959801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.960572 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.960575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:02.961010 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:02.961019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.961113 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.965699 (+ 4586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:02.965713 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:02.967221 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:02.968783 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:02.968794 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:02.968799 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:02.969037 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:02.969265 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:02.969268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:02.969270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:02.969317 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:02.969631 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:02.969636 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:02.969740 (+ 104us) client_negotiation.cc:769] Sending connection context
0501 19:56:02.969837 (+ 97us) client_negotiation.cc:240] Negotiation successful
0501 19:56:02.969959 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":106,"threads_started":1}
I20240501 19:56:02.970171 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:02.958469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44210 (local address 127.8.168.126:44223)
0501 19:56:02.958705 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:02.958709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:02.958781 (+ 72us) server_negotiation.cc:407] Connection header received
0501 19:56:02.958904 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:02.958907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:02.958949 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:02.959032 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:02.959895 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.960420 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:02.961101 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:02.961253 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:02.967352 (+ 6099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:02.967373 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:02.967375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:02.967397 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:02.968644 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:02.969130 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:02.969132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:02.969134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:02.969180 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:02.969396 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:02.969398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:02.969400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:02.969548 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:02.969617 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:02.969877 (+ 260us) server_negotiation.cc:299] Negotiation successful
0501 19:56:02.970020 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
I20240501 19:56:02.974356 9980 catalog_manager.cc:1904] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:44210:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240501 19:56:02.975109 9980 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:02.979791 10117 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb8961ccf844852aaa317a69b409e13 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23dde101c2174c9eaea49d91f6b9de71]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 19:56:02.979826 10257 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb8961ccf844852aaa317a69b409e13 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23dde101c2174c9eaea49d91f6b9de71]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 19:56:02.979899 10397 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb8961ccf844852aaa317a69b409e13 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23dde101c2174c9eaea49d91f6b9de71]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 19:56:02.980160 10117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb8961ccf844852aaa317a69b409e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.980180 10397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb8961ccf844852aaa317a69b409e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.980180 10257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb8961ccf844852aaa317a69b409e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:02.982803 10486 tablet_bootstrap.cc:492] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1: Bootstrap starting.
I20240501 19:56:02.983594 10486 tablet_bootstrap.cc:654] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.984059 10492 tablet_bootstrap.cc:492] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522: Bootstrap starting.
I20240501 19:56:02.984580 10486 tablet_bootstrap.cc:492] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1: No bootstrap required, opened a new log
I20240501 19:56:02.984653 10486 ts_tablet_manager.cc:1367] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:02.984827 10492 tablet_bootstrap.cc:654] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.985124 10486 raft_consensus.cc:348] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.985221 10486 raft_consensus.cc:374] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.985250 10486 raft_consensus.cc:729] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2522521cb7d34173a740a39319669da1, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.985400 10487 tablet_bootstrap.cc:492] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07: Bootstrap starting.
I20240501 19:56:02.985368 10486 consensus_queue.cc:260] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.985617 10486 ts_tablet_manager.cc:1398] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:02.985841 10505 tablet_replica.cc:439] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } } }
I20240501 19:56:02.985950 10505 tablet_replica.cc:442] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:02.986191 10487 tablet_bootstrap.cc:654] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:02.987121 10492 tablet_bootstrap.cc:492] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522: No bootstrap required, opened a new log
I20240501 19:56:02.987212 10492 ts_tablet_manager.cc:1367] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:02.987381 10487 tablet_bootstrap.cc:492] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07: No bootstrap required, opened a new log
I20240501 19:56:02.987473 10487 ts_tablet_manager.cc:1367] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:02.987645 10492 raft_consensus.cc:348] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.987762 10492 raft_consensus.cc:374] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.987797 10492 raft_consensus.cc:729] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121318f4216144fa8f268b1c0cf91522, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.987905 10492 consensus_queue.cc:260] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.987927 10487 raft_consensus.cc:348] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.988009 10487 raft_consensus.cc:374] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:02.988035 10487 raft_consensus.cc:729] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eeff5db83424480a26bea12c78d0e07, State: Initialized, Role: FOLLOWER
I20240501 19:56:02.988158 10492 ts_tablet_manager.cc:1398] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:02.988142 10494 tablet_replica.cc:439] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } } }
I20240501 19:56:02.988142 10487 consensus_queue.cc:260] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:02.988245 10494 tablet_replica.cc:442] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:02.988412 10491 tablet_replica.cc:439] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } } }
I20240501 19:56:02.988528 10491 tablet_replica.cc:442] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:02.988413 10487 ts_tablet_manager.cc:1398] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:03.043051 10505 raft_consensus.cc:482] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:03.043181 10505 raft_consensus.cc:504] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:03.043581 10505 leader_election.cc:288] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2eeff5db83424480a26bea12c78d0e07 (127.8.168.66:44521), 121318f4216144fa8f268b1c0cf91522 (127.8.168.67:34959)
I20240501 19:56:03.044101 10416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb8961ccf844852aaa317a69b409e13" candidate_uuid: "2522521cb7d34173a740a39319669da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121318f4216144fa8f268b1c0cf91522" is_pre_election: true
I20240501 19:56:03.044101 10273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb8961ccf844852aaa317a69b409e13" candidate_uuid: "2522521cb7d34173a740a39319669da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eeff5db83424480a26bea12c78d0e07" is_pre_election: true
I20240501 19:56:03.044250 10416 raft_consensus.cc:2454] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2522521cb7d34173a740a39319669da1 in term 0.
I20240501 19:56:03.044251 10273 raft_consensus.cc:2454] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2522521cb7d34173a740a39319669da1 in term 0.
I20240501 19:56:03.044546 10068 leader_election.cc:302] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2522521cb7d34173a740a39319669da1, 2eeff5db83424480a26bea12c78d0e07; no voters:
I20240501 19:56:03.044795 10505 raft_consensus.cc:2781] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:03.044867 10505 raft_consensus.cc:482] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:03.044893 10505 raft_consensus.cc:3037] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.075240 10505 raft_consensus.cc:504] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:03.075707 10505 leader_election.cc:288] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [CANDIDATE]: Term 1 election: Requested vote from peers 2eeff5db83424480a26bea12c78d0e07 (127.8.168.66:44521), 121318f4216144fa8f268b1c0cf91522 (127.8.168.67:34959)
I20240501 19:56:03.076162 10273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb8961ccf844852aaa317a69b409e13" candidate_uuid: "2522521cb7d34173a740a39319669da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eeff5db83424480a26bea12c78d0e07"
I20240501 19:56:03.076164 10416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb8961ccf844852aaa317a69b409e13" candidate_uuid: "2522521cb7d34173a740a39319669da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121318f4216144fa8f268b1c0cf91522"
I20240501 19:56:03.076289 10273 raft_consensus.cc:3037] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.076303 10416 raft_consensus.cc:3037] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.077240 10273 raft_consensus.cc:2454] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2522521cb7d34173a740a39319669da1 in term 1.
I20240501 19:56:03.077370 10416 raft_consensus.cc:2454] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2522521cb7d34173a740a39319669da1 in term 1.
I20240501 19:56:03.077585 10068 leader_election.cc:302] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2522521cb7d34173a740a39319669da1, 2eeff5db83424480a26bea12c78d0e07; no voters:
I20240501 19:56:03.077785 10505 raft_consensus.cc:2781] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:03.077872 10505 raft_consensus.cc:686] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [term 1 LEADER]: Becoming Leader. State: Replica: 2522521cb7d34173a740a39319669da1, State: Running, Role: LEADER
I20240501 19:56:03.078013 10505 consensus_queue.cc:237] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } }
I20240501 19:56:03.078647 10490 tablet_replica.cc:439] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1: TxnStatusTablet state changed. Reason: New leader 2522521cb7d34173a740a39319669da1. Latest consensus state: current_term: 1 leader_uuid: "2522521cb7d34173a740a39319669da1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } } }
I20240501 19:56:03.078737 10490 tablet_replica.cc:442] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1: This TxnStatusTablet replica's current role is: LEADER
I20240501 19:56:03.079108 10531 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240501 19:56:03.079263 10531 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240501 19:56:03.079739 9977 catalog_manager.cc:5484] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2522521cb7d34173a740a39319669da1 (127.8.168.65). New cstate: current_term: 1 leader_uuid: "2522521cb7d34173a740a39319669da1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:03.113276 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:03.110242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:37677 (local address 127.0.0.1:58338)
0501 19:56:03.110384 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:03.110413 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:03.110515 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:03.110810 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:03.110812 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:03.110819 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 19:56:03.111024 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:03.111029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.111987 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:03.111990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:03.112404 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:03.112411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.112521 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:03.112563 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 19:56:03.112972 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 19:56:03.112994 (+ 22us) client_negotiation.cc:769] Sending connection context
0501 19:56:03.113065 (+ 71us) client_negotiation.cc:240] Negotiation successful
0501 19:56:03.113144 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20240501 19:56:03.113267 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:03.110331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.8.168.65:37677)
0501 19:56:03.110460 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:03.110464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:03.110481 (+ 17us) server_negotiation.cc:407] Connection header received
0501 19:56:03.110642 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:03.110646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:03.110699 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:03.110793 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:03.111122 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.111866 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:03.112518 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.112701 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:03.112788 (+ 87us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:03.112874 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:03.112947 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:03.113084 (+ 137us) server_negotiation.cc:299] Negotiation successful
0501 19:56:03.113129 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20240501 19:56:03.115758 10273 raft_consensus.cc:1261] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07 [term 1 FOLLOWER]: Refusing update from remote peer 2522521cb7d34173a740a39319669da1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:03.115784 10416 raft_consensus.cc:1261] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522 [term 1 FOLLOWER]: Refusing update from remote peer 2522521cb7d34173a740a39319669da1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:03.116173 10505 consensus_queue.cc:1035] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:03.116346 10490 consensus_queue.cc:1035] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:03.117676 10491 tablet_replica.cc:439] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07: TxnStatusTablet state changed. Reason: New leader 2522521cb7d34173a740a39319669da1. Latest consensus state: current_term: 1 leader_uuid: "2522521cb7d34173a740a39319669da1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } } }
I20240501 19:56:03.117784 10491 tablet_replica.cc:442] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:03.118647 10494 tablet_replica.cc:439] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522: TxnStatusTablet state changed. Reason: New leader 2522521cb7d34173a740a39319669da1. Latest consensus state: current_term: 1 leader_uuid: "2522521cb7d34173a740a39319669da1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } } }
I20240501 19:56:03.118753 10494 tablet_replica.cc:442] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:03.120071 10500 mvcc.cc:205] Tried to move back new op lower bound from 7022974415318437888 to 7022974415168966656. Current Snapshot: MvccSnapshot[applied={T|T < 7022974415318437888}]
I20240501 19:56:03.121277 10497 tablet_replica.cc:439] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2522521cb7d34173a740a39319669da1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } } }
I20240501 19:56:03.121378 10497 tablet_replica.cc:442] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1: This TxnStatusTablet replica's current role is: LEADER
I20240501 19:56:03.122213 10490 tablet_replica.cc:439] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2522521cb7d34173a740a39319669da1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } } }
I20240501 19:56:03.122313 10490 tablet_replica.cc:442] T 4bb8961ccf844852aaa317a69b409e13 P 2522521cb7d34173a740a39319669da1: This TxnStatusTablet replica's current role is: LEADER
I20240501 19:56:03.122951 10491 tablet_replica.cc:439] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2522521cb7d34173a740a39319669da1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } } }
I20240501 19:56:03.123049 10491 tablet_replica.cc:442] T 4bb8961ccf844852aaa317a69b409e13 P 2eeff5db83424480a26bea12c78d0e07: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:03.123574 10494 tablet_replica.cc:439] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2522521cb7d34173a740a39319669da1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 } } peers { permanent_uuid: "2eeff5db83424480a26bea12c78d0e07" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 44521 } } peers { permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 } } }
I20240501 19:56:03.123656 10501 mvcc.cc:205] Tried to move back new op lower bound from 7022974415318437888 to 7022974415168966656. Current Snapshot: MvccSnapshot[applied={T|T < 7022974415318437888}]
I20240501 19:56:03.123664 10494 tablet_replica.cc:442] T 4bb8961ccf844852aaa317a69b409e13 P 121318f4216144fa8f268b1c0cf91522: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:03.129793 10538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:03.126757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.8.168.66:44521)
0501 19:56:03.127004 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:03.127008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:03.127019 (+ 11us) server_negotiation.cc:407] Connection header received
0501 19:56:03.127180 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:03.127183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:03.127235 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:03.127340 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:03.127660 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.128140 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:03.128777 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.128944 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:03.128996 (+ 52us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:03.129429 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:03.129536 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:03.129617 (+ 81us) server_negotiation.cc:299] Negotiation successful
0501 19:56:03.129660 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":99,"threads_started":1}
I20240501 19:56:03.138531 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:03.135713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:43165 (local address 127.8.168.65:37677)
0501 19:56:03.135863 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:03.135866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:03.136058 (+ 192us) server_negotiation.cc:407] Connection header received
0501 19:56:03.136171 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:03.136174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:03.136214 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:03.136282 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:03.136663 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.137144 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:03.137980 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.138108 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:03.138141 (+ 33us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:03.138201 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:03.138253 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:03.138397 (+ 144us) server_negotiation.cc:299] Negotiation successful
0501 19:56:03.138436 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20240501 19:56:03.138597 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:03.135601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:37677 (local address 127.8.168.66:43165)
0501 19:56:03.135978 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:03.135991 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:03.136089 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:03.136301 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:03.136305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:03.136311 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 19:56:03.136577 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:03.136583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.137234 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:03.137238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:03.137884 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:03.137893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.138015 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:03.138054 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 19:56:03.138302 (+ 248us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 19:56:03.138308 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 19:56:03.138363 (+ 55us) client_negotiation.cc:240] Negotiation successful
0501 19:56:03.138424 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":97,"threads_started":1}
I20240501 19:56:03.145187 10416 raft_consensus.cc:3037] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.145424 10137 raft_consensus.cc:1261] T a9052c6dab394e03b88ee44058a88cae P 2522521cb7d34173a740a39319669da1 [term 1 FOLLOWER]: Refusing update from remote peer 2eeff5db83424480a26bea12c78d0e07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:03.145892 10518 consensus_queue.cc:1035] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2522521cb7d34173a740a39319669da1" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:03.165791 10544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:03.162521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:44521 (local address 127.8.168.65:46047)
0501 19:56:03.162833 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:03.162847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:03.162951 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:03.163157 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:03.163160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:03.163167 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 19:56:03.163487 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:03.163495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.164434 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:03.164437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:03.164997 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:03.165006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.165104 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:03.165196 (+ 92us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 19:56:03.165513 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 19:56:03.165519 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 19:56:03.165574 (+ 55us) client_negotiation.cc:240] Negotiation successful
0501 19:56:03.165642 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":119,"threads_started":1}
I20240501 19:56:03.166441 10538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:03.162592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:46047 (local address 127.8.168.66:44521)
0501 19:56:03.162706 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:03.162710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:03.162916 (+ 206us) server_negotiation.cc:407] Connection header received
0501 19:56:03.163036 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:03.163039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:03.163079 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:03.163136 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:03.163601 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.164320 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:03.165103 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.165312 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:03.165363 (+ 51us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:03.165426 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:03.165486 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:03.165594 (+ 108us) server_negotiation.cc:299] Negotiation successful
0501 19:56:03.165639 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240501 19:56:03.177369 10416 raft_consensus.cc:1261] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522 [term 1 FOLLOWER]: Refusing update from remote peer 2eeff5db83424480a26bea12c78d0e07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:03.178141 10491 consensus_queue.cc:1035] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "121318f4216144fa8f268b1c0cf91522" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:03.193339 9977 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44196:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:03.193512 9977 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44196:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:03.195956 9977 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 3532ee91f68948f88709952d2f63e643: Sending DeleteTablet for 3 replicas of tablet a9052c6dab394e03b88ee44058a88cae
I20240501 19:56:03.197273 10117 tablet_service.cc:1514] Processing DeleteTablet for tablet a9052c6dab394e03b88ee44058a88cae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:03 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:58320
I20240501 19:56:03.197635 10512 tablet_replica.cc:330] T a9052c6dab394e03b88ee44058a88cae P 2522521cb7d34173a740a39319669da1: stopping tablet replica
I20240501 19:56:03.197898 10512 raft_consensus.cc:2229] T a9052c6dab394e03b88ee44058a88cae P 2522521cb7d34173a740a39319669da1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:03.197995 10397 tablet_service.cc:1514] Processing DeleteTablet for tablet a9052c6dab394e03b88ee44058a88cae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:03 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:44062
I20240501 19:56:03.198287 10512 raft_consensus.cc:2258] T a9052c6dab394e03b88ee44058a88cae P 2522521cb7d34173a740a39319669da1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:03.198428 10510 tablet_replica.cc:330] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522: stopping tablet replica
I20240501 19:56:03.198480 10257 tablet_service.cc:1514] Processing DeleteTablet for tablet a9052c6dab394e03b88ee44058a88cae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:03 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:51606
I20240501 19:56:03.198612 10510 raft_consensus.cc:2229] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:03.198829 10511 tablet_replica.cc:330] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07: stopping tablet replica
I20240501 19:56:03.199018 10510 raft_consensus.cc:2258] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:03.199124 10511 raft_consensus.cc:2229] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:03.199880 10511 raft_consensus.cc:2258] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:03.199930 10512 ts_tablet_manager.cc:1875] T a9052c6dab394e03b88ee44058a88cae P 2522521cb7d34173a740a39319669da1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:03.201509 10511 ts_tablet_manager.cc:1875] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:03.205328 10510 ts_tablet_manager.cc:1875] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522: Deleting tablet data with delete state TABLET_DATA_DELETED
May 01 19:56:03 dist-test-slave-tzct krb5kdc[9935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593362, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.168.65@KRBTEST.COM
I20240501 19:56:03.224627 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:03.203387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58354 (local address 127.8.168.65:37677)
0501 19:56:03.203521 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:03.203525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:03.213107 (+ 9582us) server_negotiation.cc:407] Connection header received
0501 19:56:03.213368 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:03.213373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:03.213434 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:03.213493 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:03.214678 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.215391 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:03.216312 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.216501 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:03.220422 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:03.220456 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:03.220472 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:03.220503 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:03.222617 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:03.223105 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:03.223111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:03.223114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:03.223185 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:03.223482 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:03.223486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:03.223488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:03.223731 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:03.223922 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:03.224315 (+ 393us) server_negotiation.cc:299] Negotiation successful
0501 19:56:03.224448 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20240501 19:56:03.225872 10177 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58354
I20240501 19:56:03.236243 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:03.227593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.8.168.126:44223)
0501 19:56:03.227924 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:03.227929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:03.228053 (+ 124us) server_negotiation.cc:407] Connection header received
0501 19:56:03.228172 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:03.228175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:03.228226 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:03.228313 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:03.229268 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.230093 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:03.231012 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:03.231231 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:03.232807 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:03.232829 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:03.232832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:03.232860 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:03.234635 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:03.235114 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:03.235119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:03.235121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:03.235177 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:03.235423 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:03.235426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:03.235429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:03.235610 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:03.235738 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:03.235947 (+ 209us) server_negotiation.cc:299] Negotiation successful
0501 19:56:03.236069 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
W20240501 19:56:03.236958 9977 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44212
I20240501 19:56:03.238204 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 10052
I20240501 19:56:03.283093 10511 ts_tablet_manager.cc:1888] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 19:56:03.283196 10511 log.cc:1196] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/a9052c6dab394e03b88ee44058a88cae
W20240501 19:56:03.283372 9964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.168.65:37677 (error 108)
I20240501 19:56:03.283452 10511 ts_tablet_manager.cc:1909] T a9052c6dab394e03b88ee44058a88cae P 2eeff5db83424480a26bea12c78d0e07: Deleting consensus metadata
I20240501 19:56:03.283643 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 10192
I20240501 19:56:03.285143 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:03.284657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:37677 (local address 127.0.0.1:58366)
0501 19:56:03.284978 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:03.285039 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":92,"threads_started":1}
W20240501 19:56:03.285354 9964 catalog_manager.cc:4573] TS 2522521cb7d34173a740a39319669da1 (127.8.168.65:37677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9052c6dab394e03b88ee44058a88cae: Network error: Client connection negotiation failed: client connection to 127.8.168.65:37677: connect: Connection refused (error 111)
I20240501 19:56:03.306152 10510 ts_tablet_manager.cc:1888] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 19:56:03.306278 10510 log.cc:1196] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/a9052c6dab394e03b88ee44058a88cae
I20240501 19:56:03.306519 10510 ts_tablet_manager.cc:1909] T a9052c6dab394e03b88ee44058a88cae P 121318f4216144fa8f268b1c0cf91522: Deleting consensus metadata
I20240501 19:56:03.309953 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 10332
I20240501 19:56:03.310683 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:03.310023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:44521 (local address 127.0.0.1:51634)
0501 19:56:03.310553 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:03.310600 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.66:44521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437}
I20240501 19:56:03.332800 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 9948
2024-05-01T19:56:03Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2821 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 19:56:03 dist-test-slave-tzct krb5kdc[10559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 19:56:03 dist-test-slave-tzct krb5kdc[10559](info): set up 2 sockets
May 01 19:56:03 dist-test-slave-tzct krb5kdc[10559](info): commencing operation
krb5kdc: starting...
W20240501 19:56:04.268615 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.913s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 19:56:04 dist-test-slave-tzct krb5kdc[10559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T19:56:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 19:56:04.380667 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:41585
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:41585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:04.485945 10572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:04.486178 10572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:04.486227 10572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:04.489207 10572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:04.489259 10572 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240501 19:56:04.489281 10572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:04.489300 10572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:04.489323 10572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:04.489339 10572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:04.493422 10572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:41585
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:41585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:04.494241 10572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:04.495020 10572 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:04.502424 10578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:04.503011 10577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:04.503104 10580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:04.503680 10572 server_base.cc:1034] running on GCE node
I20240501 19:56:04.504225 10572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:04.505213 10572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:04.506382 10572 hybrid_clock.cc:648] HybridClock initialized: now 1714593364506362 us; error 33 us; skew 500 ppm
May 01 19:56:04 dist-test-slave-tzct krb5kdc[10559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593364, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:04.509689 10572 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 19:56:04.510660 10572 webserver.cc:460] Webserver started at http://127.8.168.126:46755/ using document root <none> and password file <none>
I20240501 19:56:04.511179 10572 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:04.511230 10572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:04.511379 10572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:04.513173 10572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "328441c5b38e489cad2d0a34c4129ae2"
format_stamp: "Formatted at 2024-05-01 19:56:04 on dist-test-slave-tzct"
server_key: "b34125ae1e24571bd29a303b785c3ac4"
server_key_iv: "aae486d453520109085a9e9cb5589683"
server_key_version: "encryptionkey@0"
I20240501 19:56:04.513607 10572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "328441c5b38e489cad2d0a34c4129ae2"
format_stamp: "Formatted at 2024-05-01 19:56:04 on dist-test-slave-tzct"
server_key: "b34125ae1e24571bd29a303b785c3ac4"
server_key_iv: "aae486d453520109085a9e9cb5589683"
server_key_version: "encryptionkey@0"
I20240501 19:56:04.516837 10572 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:04.518946 10587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:04.520012 10572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:04.520130 10572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "328441c5b38e489cad2d0a34c4129ae2"
format_stamp: "Formatted at 2024-05-01 19:56:04 on dist-test-slave-tzct"
server_key: "b34125ae1e24571bd29a303b785c3ac4"
server_key_iv: "aae486d453520109085a9e9cb5589683"
server_key_version: "encryptionkey@0"
I20240501 19:56:04.520234 10572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:04.541213 10572 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:04.544201 10572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:04.544366 10572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:04.552186 10572 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:41585
I20240501 19:56:04.552182 10649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:41585 every 8 connection(s)
I20240501 19:56:04.553263 10572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:56:04.555826 10650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.556238 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 10572
I20240501 19:56:04.556380 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:56:04.556648 8865 external_mini_cluster.cc:1428] Setting key 996b0f84340e7d31f8b01a11527610ee
I20240501 19:56:04.561241 10650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2: Bootstrap starting.
I20240501 19:56:04.563663 10650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2: Neither blocks nor log segments found. Creating new log.
May 01 19:56:04 dist-test-slave-tzct krb5kdc[10559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:04.564390 10650 log.cc:824] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:04.566135 10650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2: No bootstrap required, opened a new log
I20240501 19:56:04.569020 10650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328441c5b38e489cad2d0a34c4129ae2" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41585 } }
I20240501 19:56:04.569200 10650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.569262 10650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328441c5b38e489cad2d0a34c4129ae2, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.569362 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:04.557908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46740 (local address 127.8.168.126:41585)
0501 19:56:04.558342 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:04.558353 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:04.558388 (+ 35us) server_negotiation.cc:407] Connection header received
0501 19:56:04.558977 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:04.558995 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:04.559274 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:04.559606 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:04.560487 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:04.561296 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:04.562126 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:04.562323 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:04.564701 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:04.564742 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:04.564757 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:04.564802 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:04.566991 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:04.567443 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:04.567447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:04.567451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:04.567516 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:04.567729 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:04.567731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:04.567732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:04.568040 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:04.568181 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:04.568465 (+ 284us) server_negotiation.cc:299] Negotiation successful
0501 19:56:04.568688 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":124,"threads_started":1}
I20240501 19:56:04.569769 10650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328441c5b38e489cad2d0a34c4129ae2" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41585 } }
I20240501 19:56:04.569927 10650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:04.569986 10650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:04.570088 10650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.570889 10650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328441c5b38e489cad2d0a34c4129ae2" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41585 } }
I20240501 19:56:04.571190 10650 leader_election.cc:302] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 328441c5b38e489cad2d0a34c4129ae2; no voters:
I20240501 19:56:04.571449 10650 leader_election.cc:288] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:04.571513 10655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.571697 10655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [term 1 LEADER]: Becoming Leader. State: Replica: 328441c5b38e489cad2d0a34c4129ae2, State: Running, Role: LEADER
I20240501 19:56:04.571964 10655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328441c5b38e489cad2d0a34c4129ae2" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41585 } }
I20240501 19:56:04.572660 10650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:04.573480 10656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 328441c5b38e489cad2d0a34c4129ae2. Latest consensus state: current_term: 1 leader_uuid: "328441c5b38e489cad2d0a34c4129ae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328441c5b38e489cad2d0a34c4129ae2" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41585 } } }
I20240501 19:56:04.573524 10657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "328441c5b38e489cad2d0a34c4129ae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328441c5b38e489cad2d0a34c4129ae2" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41585 } } }
I20240501 19:56:04.573585 10656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:04.573799 10657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:04.573979 10663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:04.576658 10663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:04.581485 10663 catalog_manager.cc:1347] Generated new cluster ID: 64850911d44649da99356885cd83597f
I20240501 19:56:04.581554 10663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:04.599614 10663 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:04.600582 10663 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:04.609480 10663 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2: Generated new TSK 0
I20240501 19:56:04.610181 10663 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
I20240501 19:56:04.639307 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:41585
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:04.747314 10676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:04.747539 10676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:04.747613 10676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:04.750672 10676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:04.750737 10676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:04.750808 10676 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240501 19:56:04.750839 10676 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 19:56:04.750890 10676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:56:04.755103 10676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.168.126:41585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:04.756006 10676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:04.756846 10676 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:04.764328 10682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:04.764886 10681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:04.765424 10684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:04.765870 10676 server_base.cc:1034] running on GCE node
I20240501 19:56:04.766180 10676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:04.766672 10676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:04.767849 10676 hybrid_clock.cc:648] HybridClock initialized: now 1714593364767834 us; error 29 us; skew 500 ppm
May 01 19:56:04 dist-test-slave-tzct krb5kdc[10559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593364, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:04.770380 10676 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 19:56:04.771266 10676 webserver.cc:460] Webserver started at http://127.8.168.65:32779/ using document root <none> and password file <none>
I20240501 19:56:04.771761 10676 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:04.771822 10676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:04.771982 10676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:04.773803 10676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "d38e614bf7c24d979ba5385e5074ba8b"
format_stamp: "Formatted at 2024-05-01 19:56:04 on dist-test-slave-tzct"
server_key: "79c7d9ee91788607032df91b0d5506e1"
server_key_iv: "53693a276c979b3de11de75530958894"
server_key_version: "encryptionkey@0"
I20240501 19:56:04.774281 10676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "d38e614bf7c24d979ba5385e5074ba8b"
format_stamp: "Formatted at 2024-05-01 19:56:04 on dist-test-slave-tzct"
server_key: "79c7d9ee91788607032df91b0d5506e1"
server_key_iv: "53693a276c979b3de11de75530958894"
server_key_version: "encryptionkey@0"
I20240501 19:56:04.777531 10676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:04.779702 10691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:04.780675 10676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:56:04.780831 10676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "d38e614bf7c24d979ba5385e5074ba8b"
format_stamp: "Formatted at 2024-05-01 19:56:04 on dist-test-slave-tzct"
server_key: "79c7d9ee91788607032df91b0d5506e1"
server_key_iv: "53693a276c979b3de11de75530958894"
server_key_version: "encryptionkey@0"
I20240501 19:56:04.780926 10676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:04.805610 10676 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:04.808508 10676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:04.808656 10676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:04.810032 10676 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:04.810083 10676 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:04.810137 10676 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:04.810164 10676 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 19:56:04 dist-test-slave-tzct krb5kdc[10559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593364, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:04.819864 10676 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:42745
I20240501 19:56:04.819906 10806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:42745 every 8 connection(s)
I20240501 19:56:04.820709 10676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 19:56:04.822018 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:04.810770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:57761 (local address 127.8.168.126:41585)
0501 19:56:04.810886 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:04.810889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:04.811637 (+ 748us) server_negotiation.cc:407] Connection header received
0501 19:56:04.812412 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:04.812416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:04.812491 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:04.812562 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:04.813836 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:04.814540 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:04.815215 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:04.815445 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:04.818122 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:04.818143 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:04.818147 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:04.818178 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:04.819882 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:04.820388 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:04.820392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:04.820394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:04.820450 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:04.820860 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:04.820863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:04.820865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:04.821042 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:04.821145 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:04.821700 (+ 555us) server_negotiation.cc:299] Negotiation successful
0501 19:56:04.821837 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240501 19:56:04.822595 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:04.811069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41585 (local address 127.8.168.65:57761)
0501 19:56:04.811507 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:04.811540 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:04.812214 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:04.812773 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:04.812783 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:04.813146 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:04.813692 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:04.813706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:04.814676 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:04.814680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:04.815103 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:04.815112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:04.815316 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:04.815949 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:04.815982 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:04.817952 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:04.820027 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:04.820032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:04.820040 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:04.820279 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:04.820535 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:04.820538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:04.820540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:04.820629 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:04.821155 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:04.821161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:04.821447 (+ 286us) client_negotiation.cc:769] Sending connection context
0501 19:56:04.821627 (+ 180us) client_negotiation.cc:240] Negotiation successful
0501 19:56:04.821892 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":138,"threads_started":1}
I20240501 19:56:04.823876 10807 heartbeater.cc:347] Connected to a master server at 127.8.168.126:41585
I20240501 19:56:04.824028 10807 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:04.824532 10807 heartbeater.cc:511] Master 127.8.168.126:41585 requested a full tablet report, sending...
I20240501 19:56:04.824939 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 10676
I20240501 19:56:04.825068 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 19:56:04.825330 8865 external_mini_cluster.cc:1428] Setting key 53edf3c4bb52ac2d2907d331277f2ccb
I20240501 19:56:04.826064 10603 ts_manager.cc:194] Registered new tserver with Master: d38e614bf7c24d979ba5385e5074ba8b (127.8.168.65:42745)
I20240501 19:56:04.827443 10603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:57761
I20240501 19:56:04.834143 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:04.825795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41585 (local address 127.8.168.65:51939)
0501 19:56:04.826053 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:04.826067 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:04.826145 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:04.828318 (+ 2173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:04.828321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:04.828466 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:04.828771 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:04.828780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:04.829577 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:04.829580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:04.830073 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:04.830083 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:04.830190 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:04.830684 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:04.830698 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:04.831048 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:04.832827 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:04.832831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:04.832833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:04.832998 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:04.833255 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:04.833258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:04.833259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:04.833315 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:04.833683 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:04.833686 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:04.833804 (+ 118us) client_negotiation.cc:769] Sending connection context
0501 19:56:04.833894 (+ 90us) client_negotiation.cc:240] Negotiation successful
0501 19:56:04.833996 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":113,"threads_started":1}
I20240501 19:56:04.834223 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:04.828008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:51939 (local address 127.8.168.126:41585)
0501 19:56:04.828115 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:04.828119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:04.828132 (+ 13us) server_negotiation.cc:407] Connection header received
0501 19:56:04.828165 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:04.828168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:04.828215 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:04.828281 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:04.828893 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:04.829467 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:04.830171 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:04.830301 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:04.831170 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:04.831183 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:04.831185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:04.831205 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:04.832675 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:04.833090 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:04.833094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:04.833096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:04.833143 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:04.833400 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:04.833403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:04.833405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:04.833562 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:04.833662 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:04.833969 (+ 307us) server_negotiation.cc:299] Negotiation successful
0501 19:56:04.834074 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for kudu/127.8.168.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
I20240501 19:56:04.851477 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:41585
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:04.956243 10816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:04.956459 10816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:04.956523 10816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:04.959491 10816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:04.959549 10816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:04.959600 10816 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240501 19:56:04.959628 10816 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 19:56:04.959687 10816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:56:04.963966 10816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.168.126:41585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:04.964906 10816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:04.965739 10816 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:04.973951 10821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:04.973959 10822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:04.974537 10824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:04.975003 10816 server_base.cc:1034] running on GCE node
I20240501 19:56:04.975374 10816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:04.975957 10816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:04.977110 10816 hybrid_clock.cc:648] HybridClock initialized: now 1714593364977092 us; error 32 us; skew 500 ppm
May 01 19:56:04 dist-test-slave-tzct krb5kdc[10559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593364, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:04.979624 10816 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 19:56:04.980484 10816 webserver.cc:460] Webserver started at http://127.8.168.66:42273/ using document root <none> and password file <none>
I20240501 19:56:04.981038 10816 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:04.981096 10816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:04.981257 10816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:04.982997 10816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "5e7f6611c0e84cdab59c4063b64ab205"
format_stamp: "Formatted at 2024-05-01 19:56:04 on dist-test-slave-tzct"
server_key: "5705d09b1f53bc1713059267d3a89b41"
server_key_iv: "12738f8525f2f64bcb9265c2caa77930"
server_key_version: "encryptionkey@0"
I20240501 19:56:04.983461 10816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "5e7f6611c0e84cdab59c4063b64ab205"
format_stamp: "Formatted at 2024-05-01 19:56:04 on dist-test-slave-tzct"
server_key: "5705d09b1f53bc1713059267d3a89b41"
server_key_iv: "12738f8525f2f64bcb9265c2caa77930"
server_key_version: "encryptionkey@0"
I20240501 19:56:04.986869 10816 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:04.989070 10831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:04.990080 10816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:56:04.990186 10816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "5e7f6611c0e84cdab59c4063b64ab205"
format_stamp: "Formatted at 2024-05-01 19:56:04 on dist-test-slave-tzct"
server_key: "5705d09b1f53bc1713059267d3a89b41"
server_key_iv: "12738f8525f2f64bcb9265c2caa77930"
server_key_version: "encryptionkey@0"
I20240501 19:56:04.990272 10816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:05.009068 10816 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:05.011976 10816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:05.012161 10816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:05.013747 10816 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:05.013808 10816 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:05.013866 10816 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:05.013933 10816 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 19:56:05 dist-test-slave-tzct krb5kdc[10559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593364, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:05.025080 10816 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:45073
I20240501 19:56:05.025115 10946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:45073 every 8 connection(s)
I20240501 19:56:05.026250 10816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 19:56:05.026471 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.014371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:51489 (local address 127.8.168.126:41585)
0501 19:56:05.014516 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.014521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.015255 (+ 734us) server_negotiation.cc:407] Connection header received
0501 19:56:05.016091 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.016095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.016160 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.016296 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:05.017750 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.018511 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.019215 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.019432 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.022165 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:05.022202 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:05.022206 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:05.022242 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:05.024233 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:05.024896 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:05.024901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:05.024903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:05.024971 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:05.025305 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:05.025308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:05.025310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:05.025523 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:05.025630 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.026152 (+ 522us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.026296 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20240501 19:56:05.026937 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 10816
I20240501 19:56:05.027058 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 19:56:05.027009 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.014667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41585 (local address 127.8.168.66:51489)
0501 19:56:05.015098 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.015141 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.015878 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.016479 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.016489 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.017022 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:05.017598 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.017611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.018633 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.018636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.019091 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.019101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.019325 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.020040 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:05.020082 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:05.021988 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:05.024357 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:05.024362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:05.024370 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:05.024703 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:05.025087 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:05.025091 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:05.025093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:05.025206 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:05.025620 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:05.025624 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:05.025883 (+ 259us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.026046 (+ 163us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.026271 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20240501 19:56:05.027377 8865 external_mini_cluster.cc:1428] Setting key 7d2ffab13579963d392fb84df982b16b
I20240501 19:56:05.028366 10947 heartbeater.cc:347] Connected to a master server at 127.8.168.126:41585
I20240501 19:56:05.028556 10947 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:05.029214 10947 heartbeater.cc:511] Master 127.8.168.126:41585 requested a full tablet report, sending...
I20240501 19:56:05.030678 10604 ts_manager.cc:194] Registered new tserver with Master: 5e7f6611c0e84cdab59c4063b64ab205 (127.8.168.66:45073)
I20240501 19:56:05.031311 10604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:51489
WARNING: no policy specified for kudu/127.8.168.67@KRBTEST.COM; defaulting to no policy
I20240501 19:56:05.038188 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.031578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:40637 (local address 127.8.168.126:41585)
0501 19:56:05.031744 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.031747 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.031760 (+ 13us) server_negotiation.cc:407] Connection header received
0501 19:56:05.031791 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.031794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.031841 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.031925 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:05.032580 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.033324 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.033982 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.034164 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.035110 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:05.035125 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:05.035127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:05.035147 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:05.036792 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:05.037152 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:05.037156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:05.037158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:05.037205 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:05.037421 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:05.037425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:05.037427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:05.037589 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:05.037675 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.037921 (+ 246us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.038026 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20240501 19:56:05.038208 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.030539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41585 (local address 127.8.168.66:40637)
0501 19:56:05.030823 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.030837 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.030918 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.031969 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.031972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.032167 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:05.032432 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.032458 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.033441 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.033445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.033873 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.033881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.033985 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.034645 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:05.034657 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:05.034983 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:05.036901 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:05.036905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:05.036907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:05.037065 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:05.037297 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:05.037299 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:05.037301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:05.037344 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:05.037676 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:05.037679 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:05.037756 (+ 77us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.037856 (+ 100us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.037987 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
I20240501 19:56:05.054224 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:41585
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:05.161032 10956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:05.161252 10956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:05.161320 10956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:05.164304 10956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:05.164362 10956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:05.164414 10956 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240501 19:56:05.164444 10956 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 19:56:05.164511 10956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:56:05.168550 10956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.168.126:41585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:05.169401 10956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:05.170199 10956 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:05.177767 10961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:05.178318 10956 server_base.cc:1034] running on GCE node
W20240501 19:56:05.178805 10962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:05.179230 10964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:05.179569 10956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:05.180063 10956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:05.181205 10956 hybrid_clock.cc:648] HybridClock initialized: now 1714593365181191 us; error 26 us; skew 500 ppm
May 01 19:56:05 dist-test-slave-tzct krb5kdc[10559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593365, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:05.183768 10956 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 19:56:05.184695 10956 webserver.cc:460] Webserver started at http://127.8.168.67:42707/ using document root <none> and password file <none>
I20240501 19:56:05.185248 10956 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:05.185300 10956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:05.185478 10956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:05.187222 10956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "a0e01564d00b4073a181140b862fd7f4"
format_stamp: "Formatted at 2024-05-01 19:56:05 on dist-test-slave-tzct"
server_key: "28101f0d36587e0b2f857ffb847d491d"
server_key_iv: "02ea28552a5901b62aabda44fa4788d4"
server_key_version: "encryptionkey@0"
I20240501 19:56:05.187676 10956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "a0e01564d00b4073a181140b862fd7f4"
format_stamp: "Formatted at 2024-05-01 19:56:05 on dist-test-slave-tzct"
server_key: "28101f0d36587e0b2f857ffb847d491d"
server_key_iv: "02ea28552a5901b62aabda44fa4788d4"
server_key_version: "encryptionkey@0"
I20240501 19:56:05.190960 10956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:05.193228 10971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:05.194222 10956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:05.194393 10956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "a0e01564d00b4073a181140b862fd7f4"
format_stamp: "Formatted at 2024-05-01 19:56:05 on dist-test-slave-tzct"
server_key: "28101f0d36587e0b2f857ffb847d491d"
server_key_iv: "02ea28552a5901b62aabda44fa4788d4"
server_key_version: "encryptionkey@0"
I20240501 19:56:05.194490 10956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:05.214521 10956 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:05.217157 10956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:05.217305 10956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:05.218586 10956 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:05.218638 10956 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:05.218690 10956 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:05.218722 10956 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 19:56:05 dist-test-slave-tzct krb5kdc[10559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593365, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:05.228051 10956 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:44535
I20240501 19:56:05.228088 11086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:44535 every 8 connection(s)
I20240501 19:56:05.229096 10956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 19:56:05.229496 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 10956
I20240501 19:56:05.229600 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 19:56:05.229847 8865 external_mini_cluster.cc:1428] Setting key 023a35271c72542105af55d1ae576337
I20240501 19:56:05.230701 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.219223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:48105 (local address 127.8.168.126:41585)
0501 19:56:05.219378 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.219383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.220038 (+ 655us) server_negotiation.cc:407] Connection header received
0501 19:56:05.220889 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.220893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.220951 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.221025 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:05.222624 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.223425 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.224195 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.224405 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.227220 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:05.227240 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:05.227242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:05.227269 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:05.228827 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:05.229298 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:05.229301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:05.229304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:05.229354 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:05.229651 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:05.229654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:05.229656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:05.229819 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:05.229908 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.230420 (+ 512us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.230549 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20240501 19:56:05.231356 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.219482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41585 (local address 127.8.168.67:48105)
0501 19:56:05.219892 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.219932 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.220652 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.221301 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.221309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.221775 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:05.222453 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.222479 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.223572 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.223576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.224084 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.224093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.224300 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.225009 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:05.225044 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:05.227063 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:05.228947 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:05.228953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:05.228963 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:05.229199 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:05.229434 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:05.229437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:05.229439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:05.229549 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:05.229919 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:05.229926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:05.230166 (+ 240us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.230367 (+ 201us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.230620 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":104,"threads_started":1}
I20240501 19:56:05.232331 11087 heartbeater.cc:347] Connected to a master server at 127.8.168.126:41585
I20240501 19:56:05.232466 11087 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:05.233139 11087 heartbeater.cc:511] Master 127.8.168.126:41585 requested a full tablet report, sending...
I20240501 19:56:05.234071 10603 ts_manager.cc:194] Registered new tserver with Master: a0e01564d00b4073a181140b862fd7f4 (127.8.168.67:44535)
I20240501 19:56:05.234619 10603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:48105
I20240501 19:56:05.240010 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.234291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41585 (local address 127.8.168.67:54037)
0501 19:56:05.234588 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.234599 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.234683 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.234905 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.234909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.235096 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:05.235326 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.235332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.236282 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.236285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.236643 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.236650 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.236759 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.237241 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:05.237253 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:05.237500 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:05.239048 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:05.239051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:05.239053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:05.239198 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:05.239371 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:05.239373 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:05.239374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:05.239409 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:05.239634 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:05.239636 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:05.239697 (+ 61us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.239784 (+ 87us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.239876 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
I20240501 19:56:05.240021 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.234271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:54037 (local address 127.8.168.126:41585)
0501 19:56:05.234368 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.234371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.234661 (+ 290us) server_negotiation.cc:407] Connection header received
0501 19:56:05.234764 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.234767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.234809 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.234879 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:05.235439 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.236168 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.236757 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.236889 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.237626 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:05.237642 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:05.237644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:05.237668 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:05.238953 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:05.239278 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:05.239280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:05.239282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:05.239314 (+ 32us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:05.239462 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:05.239464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:05.239465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:05.239576 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:05.239639 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.239841 (+ 202us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.239918 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240501 19:56:05.243408 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 01 19:56:05 dist-test-slave-tzct krb5kdc[10559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 01 19:56:05 dist-test-slave-tzct krb5kdc[10559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:05.265332 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.258347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46756 (local address 127.8.168.126:41585)
0501 19:56:05.258489 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.258493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.258508 (+ 15us) server_negotiation.cc:407] Connection header received
0501 19:56:05.258553 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.258555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.258601 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.258684 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:05.259471 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.260020 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.260617 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.260828 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.262759 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:05.262779 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:05.262782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:05.262809 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:05.264054 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:05.264374 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:05.264377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:05.264378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:05.264423 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:05.264584 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:05.264586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:05.264587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:05.264811 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:05.264926 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.265106 (+ 180us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.265201 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20240501 19:56:05.267988 10603 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:56:05.270447 10603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:05.282403 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.278218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:42745 (local address 127.0.0.1:45424)
0501 19:56:05.279047 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.279091 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.279211 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.279861 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.279866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.279888 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:05.280176 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.280185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.281244 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.281247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.281956 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.281964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.282057 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.282117 (+ 60us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.282210 (+ 93us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.282294 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":80,"threads_started":1}
I20240501 19:56:05.283170 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.278500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:45073 (local address 127.0.0.1:47942)
0501 19:56:05.279435 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.279447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.279522 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.280177 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.280180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.280198 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:05.280410 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.280419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.281830 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.281834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.282780 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.282790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.282898 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.282914 (+ 16us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.282960 (+ 46us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.283013 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"thread_start_us":80,"threads_started":1}
I20240501 19:56:05.283644 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.278364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45424 (local address 127.8.168.65:42745)
0501 19:56:05.279113 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.279120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.279146 (+ 26us) server_negotiation.cc:407] Connection header received
0501 19:56:05.279530 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.279536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.279707 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.279849 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:05.280308 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.281139 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.282708 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.283218 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.283336 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.283407 (+ 71us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.283523 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":83,"threads_started":1}
I20240501 19:56:05.283761 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.278972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58966 (local address 127.8.168.67:44535)
0501 19:56:05.279729 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.279734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.279752 (+ 18us) server_negotiation.cc:407] Connection header received
0501 19:56:05.279822 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.279827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.279947 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.280070 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:05.280430 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.281723 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.282918 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.283425 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.283514 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.283575 (+ 61us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.283650 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":99,"threads_started":1}
I20240501 19:56:05.284282 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.278737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.8.168.66:45073)
0501 19:56:05.279413 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.279419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.279617 (+ 198us) server_negotiation.cc:407] Connection header received
0501 19:56:05.279708 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.279713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.279884 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.280019 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:05.280534 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.281696 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.283440 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.283925 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.284019 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.284081 (+ 62us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.284157 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":98,"threads_started":1}
I20240501 19:56:05.283170 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.278297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:44535 (local address 127.0.0.1:58966)
0501 19:56:05.279193 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.279205 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.279481 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.280069 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.280072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.280088 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:05.280322 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.280332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.281830 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.281834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.282779 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.282790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.282897 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.282914 (+ 17us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.282960 (+ 46us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.283013 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"thread_start_us":107,"threads_started":1}
I20240501 19:56:05.286211 10741 tablet_service.cc:1467] Processing CreateTablet for tablet ee6b06647ffe49b1a19a7921f219b20a (DEFAULT_TABLE table=test-table [id=a88cdec672a44b1aa48c6b7e93a8e4d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.286654 11021 tablet_service.cc:1467] Processing CreateTablet for tablet ee6b06647ffe49b1a19a7921f219b20a (DEFAULT_TABLE table=test-table [id=a88cdec672a44b1aa48c6b7e93a8e4d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.286667 10881 tablet_service.cc:1467] Processing CreateTablet for tablet ee6b06647ffe49b1a19a7921f219b20a (DEFAULT_TABLE table=test-table [id=a88cdec672a44b1aa48c6b7e93a8e4d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.287070 10741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee6b06647ffe49b1a19a7921f219b20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.287532 10881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee6b06647ffe49b1a19a7921f219b20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.287532 11021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee6b06647ffe49b1a19a7921f219b20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.292135 11110 tablet_bootstrap.cc:492] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b: Bootstrap starting.
I20240501 19:56:05.293432 11111 tablet_bootstrap.cc:492] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4: Bootstrap starting.
I20240501 19:56:05.295306 11110 tablet_bootstrap.cc:654] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.295776 11111 tablet_bootstrap.cc:654] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.296113 11110 log.cc:824] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:05.296501 11111 log.cc:824] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:05.296838 11112 tablet_bootstrap.cc:492] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205: Bootstrap starting.
I20240501 19:56:05.298004 11110 tablet_bootstrap.cc:492] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b: No bootstrap required, opened a new log
I20240501 19:56:05.298290 11110 ts_tablet_manager.cc:1367] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 19:56:05.298696 11111 tablet_bootstrap.cc:492] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4: No bootstrap required, opened a new log
I20240501 19:56:05.298875 11111 ts_tablet_manager.cc:1367] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 19:56:05.299247 11112 tablet_bootstrap.cc:654] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.299997 11112 log.cc:824] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:05.300982 11110 raft_consensus.cc:348] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } }
I20240501 19:56:05.301153 11110 raft_consensus.cc:374] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.301201 11110 raft_consensus.cc:729] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38e614bf7c24d979ba5385e5074ba8b, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.301565 11110 consensus_queue.cc:260] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } }
I20240501 19:56:05.301791 11112 tablet_bootstrap.cc:492] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205: No bootstrap required, opened a new log
I20240501 19:56:05.301957 11112 ts_tablet_manager.cc:1367] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:05.301939 11111 raft_consensus.cc:348] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } }
I20240501 19:56:05.302130 11111 raft_consensus.cc:374] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.302188 11111 raft_consensus.cc:729] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e01564d00b4073a181140b862fd7f4, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.302321 10807 heartbeater.cc:503] Master 127.8.168.126:41585 was elected leader, sending a full tablet report...
I20240501 19:56:05.302356 11110 ts_tablet_manager.cc:1398] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:05.302582 11111 consensus_queue.cc:260] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } }
I20240501 19:56:05.303375 11111 ts_tablet_manager.cc:1398] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:56:05.303517 11087 heartbeater.cc:503] Master 127.8.168.126:41585 was elected leader, sending a full tablet report...
I20240501 19:56:05.306237 11112 raft_consensus.cc:348] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } }
I20240501 19:56:05.306452 11112 raft_consensus.cc:374] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.306515 11112 raft_consensus.cc:729] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7f6611c0e84cdab59c4063b64ab205, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.306883 11112 consensus_queue.cc:260] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } }
I20240501 19:56:05.307452 10947 heartbeater.cc:503] Master 127.8.168.126:41585 was elected leader, sending a full tablet report...
I20240501 19:56:05.307555 11112 ts_tablet_manager.cc:1398] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20240501 19:56:05.323547 10808 tablet.cc:2369] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:05.359195 11116 raft_consensus.cc:482] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:05.359411 11116 raft_consensus.cc:504] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } }
I20240501 19:56:05.360502 11116 leader_election.cc:288] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0e01564d00b4073a181140b862fd7f4 (127.8.168.67:44535), 5e7f6611c0e84cdab59c4063b64ab205 (127.8.168.66:45073)
I20240501 19:56:05.364012 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.360753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:44535 (local address 127.8.168.65:57637)
0501 19:56:05.361174 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.361190 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.361313 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.361555 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.361558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.361573 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:05.361824 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.361830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.362641 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.362645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.363577 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.363586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.363704 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.363727 (+ 23us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.363777 (+ 50us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.363834 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":103,"threads_started":1}
I20240501 19:56:05.364012 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.360906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:45073 (local address 127.8.168.65:59769)
0501 19:56:05.361168 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.361179 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.361288 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.361641 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.361643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.361660 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:05.361926 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.361933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.362655 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.362658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.363577 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.363586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.363703 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.363733 (+ 30us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.363777 (+ 44us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.363834 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":64,"threads_started":1}
I20240501 19:56:05.364480 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.360882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:57637 (local address 127.8.168.67:44535)
0501 19:56:05.361012 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.361017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.361295 (+ 278us) server_negotiation.cc:407] Connection header received
0501 19:56:05.361419 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.361422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.361468 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.361529 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:05.361946 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.362522 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.363713 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.364223 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.364258 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.364311 (+ 53us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.364367 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20240501 19:56:05.364480 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.360949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:59769 (local address 127.8.168.66:45073)
0501 19:56:05.361087 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.361092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.361253 (+ 161us) server_negotiation.cc:407] Connection header received
0501 19:56:05.361506 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.361508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.361555 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.361619 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:05.362044 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.362567 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.363712 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.364219 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.364258 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.364319 (+ 61us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.364372 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20240501 19:56:05.365043 10901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6b06647ffe49b1a19a7921f219b20a" candidate_uuid: "d38e614bf7c24d979ba5385e5074ba8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7f6611c0e84cdab59c4063b64ab205" is_pre_election: true
I20240501 19:56:05.365049 11041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6b06647ffe49b1a19a7921f219b20a" candidate_uuid: "d38e614bf7c24d979ba5385e5074ba8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e01564d00b4073a181140b862fd7f4" is_pre_election: true
I20240501 19:56:05.365314 10901 raft_consensus.cc:2454] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d38e614bf7c24d979ba5385e5074ba8b in term 0.
I20240501 19:56:05.365315 11041 raft_consensus.cc:2454] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d38e614bf7c24d979ba5385e5074ba8b in term 0.
I20240501 19:56:05.365792 10693 leader_election.cc:302] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0e01564d00b4073a181140b862fd7f4, d38e614bf7c24d979ba5385e5074ba8b; no voters:
I20240501 19:56:05.366101 11116 raft_consensus.cc:2781] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:05.366189 11116 raft_consensus.cc:482] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:05.366221 11116 raft_consensus.cc:3037] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.367185 11116 raft_consensus.cc:504] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } }
I20240501 19:56:05.367535 11116 leader_election.cc:288] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [CANDIDATE]: Term 1 election: Requested vote from peers a0e01564d00b4073a181140b862fd7f4 (127.8.168.67:44535), 5e7f6611c0e84cdab59c4063b64ab205 (127.8.168.66:45073)
I20240501 19:56:05.367870 11041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6b06647ffe49b1a19a7921f219b20a" candidate_uuid: "d38e614bf7c24d979ba5385e5074ba8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e01564d00b4073a181140b862fd7f4"
I20240501 19:56:05.367925 10901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6b06647ffe49b1a19a7921f219b20a" candidate_uuid: "d38e614bf7c24d979ba5385e5074ba8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7f6611c0e84cdab59c4063b64ab205"
I20240501 19:56:05.368007 11041 raft_consensus.cc:3037] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.368042 10901 raft_consensus.cc:3037] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.369232 10901 raft_consensus.cc:2454] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d38e614bf7c24d979ba5385e5074ba8b in term 1.
I20240501 19:56:05.369235 11041 raft_consensus.cc:2454] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d38e614bf7c24d979ba5385e5074ba8b in term 1.
I20240501 19:56:05.369530 10693 leader_election.cc:302] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0e01564d00b4073a181140b862fd7f4, d38e614bf7c24d979ba5385e5074ba8b; no voters:
I20240501 19:56:05.369709 11116 raft_consensus.cc:2781] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.369917 11116 raft_consensus.cc:686] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [term 1 LEADER]: Becoming Leader. State: Replica: d38e614bf7c24d979ba5385e5074ba8b, State: Running, Role: LEADER
I20240501 19:56:05.370200 11116 consensus_queue.cc:237] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } }
I20240501 19:56:05.373220 10604 catalog_manager.cc:5484] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b reported cstate change: term changed from 0 to 1, leader changed from <none> to d38e614bf7c24d979ba5385e5074ba8b (127.8.168.65). New cstate: current_term: 1 leader_uuid: "d38e614bf7c24d979ba5385e5074ba8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } health_report { overall_health: UNKNOWN } } }
May 01 19:56:05 dist-test-slave-tzct krb5kdc[10559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593364, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:05.417701 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.410437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41585 (local address 127.0.0.1:46760)
0501 19:56:05.410718 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.410729 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.410872 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.411106 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.411109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.411543 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:05.411730 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.411737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.412422 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.412425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.412850 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.412859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.412964 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.413543 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:05.413555 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:05.414955 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:05.416505 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:05.416512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:05.416515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:05.416754 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:05.416924 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:05.416925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:05.416927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:05.416968 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:05.417231 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:05.417235 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:05.417334 (+ 99us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.417423 (+ 89us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.417534 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":121,"threads_started":1}
I20240501 19:56:05.417714 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.410494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.8.168.126:41585)
0501 19:56:05.410625 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.410630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.410817 (+ 187us) server_negotiation.cc:407] Connection header received
0501 19:56:05.410965 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.410968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.411007 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.411070 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:05.411848 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.412327 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.412950 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.413108 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.415076 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:05.415092 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:05.415094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:05.415116 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:05.416402 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:05.416821 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:05.416823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:05.416825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:05.416861 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:05.417022 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:05.417024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:05.417025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:05.417148 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:05.417213 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.417461 (+ 248us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.417589 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":42}
I20240501 19:56:05.421310 10604 catalog_manager.cc:1904] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:46760:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240501 19:56:05.421965 10604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:05.426539 10881 tablet_service.cc:1467] Processing CreateTablet for tablet 42cb8c07a9414030af87e3263d8c90e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1cb68bbaada84ce7b11eee4e7919b8bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 19:56:05.426544 10741 tablet_service.cc:1467] Processing CreateTablet for tablet 42cb8c07a9414030af87e3263d8c90e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1cb68bbaada84ce7b11eee4e7919b8bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 19:56:05.426820 11021 tablet_service.cc:1467] Processing CreateTablet for tablet 42cb8c07a9414030af87e3263d8c90e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1cb68bbaada84ce7b11eee4e7919b8bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 19:56:05.427170 10741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42cb8c07a9414030af87e3263d8c90e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.427170 10881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42cb8c07a9414030af87e3263d8c90e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.427350 11021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42cb8c07a9414030af87e3263d8c90e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.429594 11111 tablet_bootstrap.cc:492] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4: Bootstrap starting.
I20240501 19:56:05.429738 11112 tablet_bootstrap.cc:492] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205: Bootstrap starting.
I20240501 19:56:05.430545 11112 tablet_bootstrap.cc:654] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.430545 11111 tablet_bootstrap.cc:654] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.431380 11110 tablet_bootstrap.cc:492] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b: Bootstrap starting.
I20240501 19:56:05.431968 11112 tablet_bootstrap.cc:492] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205: No bootstrap required, opened a new log
I20240501 19:56:05.432025 11112 ts_tablet_manager.cc:1367] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:05.432240 11111 tablet_bootstrap.cc:492] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4: No bootstrap required, opened a new log
I20240501 19:56:05.432232 11110 tablet_bootstrap.cc:654] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.432307 11111 ts_tablet_manager.cc:1367] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:05.432507 11112 raft_consensus.cc:348] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.432618 11112 raft_consensus.cc:374] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.432647 11112 raft_consensus.cc:729] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7f6611c0e84cdab59c4063b64ab205, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.432801 11111 raft_consensus.cc:348] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.432920 11111 raft_consensus.cc:374] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.432948 11111 raft_consensus.cc:729] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e01564d00b4073a181140b862fd7f4, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.433055 11111 consensus_queue.cc:260] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.433169 11112 consensus_queue.cc:260] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.433337 11117 tablet_replica.cc:439] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } }
I20240501 19:56:05.433460 11112 ts_tablet_manager.cc:1398] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:05.433466 11117 tablet_replica.cc:442] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:05.433622 11110 tablet_bootstrap.cc:492] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b: No bootstrap required, opened a new log
I20240501 19:56:05.433692 11110 ts_tablet_manager.cc:1367] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:05.433434 11118 tablet_replica.cc:439] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } }
I20240501 19:56:05.433966 11118 tablet_replica.cc:442] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:05.434137 11111 ts_tablet_manager.cc:1398] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:05.434098 11110 raft_consensus.cc:348] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.434191 11110 raft_consensus.cc:374] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.434232 11110 raft_consensus.cc:729] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38e614bf7c24d979ba5385e5074ba8b, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.434357 11110 consensus_queue.cc:260] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.434598 11110 ts_tablet_manager.cc:1398] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:05.434581 11116 tablet_replica.cc:439] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } }
I20240501 19:56:05.434680 11116 tablet_replica.cc:442] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b: This TxnStatusTablet replica's current role is: FOLLOWER
W20240501 19:56:05.479940 11088 tablet.cc:2369] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:56:05.527591 10948 tablet.cc:2369] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:05.559401 11118 raft_consensus.cc:482] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:05.559556 11118 raft_consensus.cc:504] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.560575 11118 leader_election.cc:288] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d38e614bf7c24d979ba5385e5074ba8b (127.8.168.65:42745), a0e01564d00b4073a181140b862fd7f4 (127.8.168.67:44535)
I20240501 19:56:05.563747 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.560928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:44535 (local address 127.8.168.66:44109)
0501 19:56:05.561161 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.561174 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.561261 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.561576 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.561580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.561604 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:05.561886 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.561893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.562741 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.562745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.563353 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.563361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.563462 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.563484 (+ 22us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.563532 (+ 48us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.563589 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20240501 19:56:05.564150 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.560995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:44109 (local address 127.8.168.67:44535)
0501 19:56:05.561195 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.561199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.561234 (+ 35us) server_negotiation.cc:407] Connection header received
0501 19:56:05.561382 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.561385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.561427 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.561489 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:05.562048 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.562603 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.563466 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.563897 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.563933 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.563985 (+ 52us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.564038 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20240501 19:56:05.564544 11041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb8c07a9414030af87e3263d8c90e6" candidate_uuid: "5e7f6611c0e84cdab59c4063b64ab205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e01564d00b4073a181140b862fd7f4" is_pre_election: true
I20240501 19:56:05.564689 11041 raft_consensus.cc:2454] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7f6611c0e84cdab59c4063b64ab205 in term 0.
I20240501 19:56:05.564813 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.561340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:42745 (local address 127.8.168.66:48855)
0501 19:56:05.561602 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.561612 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.561694 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.561910 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.561913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.561928 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:05.562168 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.562176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.563536 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.563539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.564421 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.564431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.564536 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.564551 (+ 15us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.564591 (+ 40us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.564644 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":88,"threads_started":1}
I20240501 19:56:05.565274 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.560960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:48855 (local address 127.8.168.65:42745)
0501 19:56:05.561085 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.561089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.561687 (+ 598us) server_negotiation.cc:407] Connection header received
0501 19:56:05.561781 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.561784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.561830 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.561892 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:05.562296 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.563408 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.564541 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.565028 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.565059 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.565101 (+ 42us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.565154 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240501 19:56:05.565275 10833 leader_election.cc:302] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7f6611c0e84cdab59c4063b64ab205, a0e01564d00b4073a181140b862fd7f4; no voters:
I20240501 19:56:05.565515 11118 raft_consensus.cc:2781] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:05.565577 11118 raft_consensus.cc:482] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:05.565606 11118 raft_consensus.cc:3037] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.565840 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb8c07a9414030af87e3263d8c90e6" candidate_uuid: "5e7f6611c0e84cdab59c4063b64ab205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38e614bf7c24d979ba5385e5074ba8b" is_pre_election: true
I20240501 19:56:05.566030 10761 raft_consensus.cc:2454] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7f6611c0e84cdab59c4063b64ab205 in term 0.
I20240501 19:56:05.566613 11118 raft_consensus.cc:504] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.566906 11118 leader_election.cc:288] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [CANDIDATE]: Term 1 election: Requested vote from peers d38e614bf7c24d979ba5385e5074ba8b (127.8.168.65:42745), a0e01564d00b4073a181140b862fd7f4 (127.8.168.67:44535)
I20240501 19:56:05.567232 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb8c07a9414030af87e3263d8c90e6" candidate_uuid: "5e7f6611c0e84cdab59c4063b64ab205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38e614bf7c24d979ba5385e5074ba8b"
I20240501 19:56:05.567271 11041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb8c07a9414030af87e3263d8c90e6" candidate_uuid: "5e7f6611c0e84cdab59c4063b64ab205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e01564d00b4073a181140b862fd7f4"
I20240501 19:56:05.567341 10761 raft_consensus.cc:3037] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.567365 11041 raft_consensus.cc:3037] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.568318 11041 raft_consensus.cc:2454] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e7f6611c0e84cdab59c4063b64ab205 in term 1.
I20240501 19:56:05.568348 10761 raft_consensus.cc:2454] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e7f6611c0e84cdab59c4063b64ab205 in term 1.
I20240501 19:56:05.568598 10832 leader_election.cc:302] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7f6611c0e84cdab59c4063b64ab205, d38e614bf7c24d979ba5385e5074ba8b; no voters:
I20240501 19:56:05.568765 11118 raft_consensus.cc:2781] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.568936 11118 raft_consensus.cc:686] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 LEADER]: Becoming Leader. State: Replica: 5e7f6611c0e84cdab59c4063b64ab205, State: Running, Role: LEADER
I20240501 19:56:05.569211 11118 consensus_queue.cc:237] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.570932 11134 tablet_replica.cc:439] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205: TxnStatusTablet state changed. Reason: New leader 5e7f6611c0e84cdab59c4063b64ab205. Latest consensus state: current_term: 1 leader_uuid: "5e7f6611c0e84cdab59c4063b64ab205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } }
I20240501 19:56:05.571034 11134 tablet_replica.cc:442] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205: This TxnStatusTablet replica's current role is: LEADER
I20240501 19:56:05.571318 11136 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240501 19:56:05.571493 11136 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240501 19:56:05.572178 10603 catalog_manager.cc:5484] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e7f6611c0e84cdab59c4063b64ab205 (127.8.168.66). New cstate: current_term: 1 leader_uuid: "5e7f6611c0e84cdab59c4063b64ab205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:05.588287 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.584915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:45073 (local address 127.0.0.1:47958)
0501 19:56:05.585135 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.585148 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.585233 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.585576 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.585579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.585586 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 19:56:05.585762 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.585767 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.586710 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.586713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.587115 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.587122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.587205 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.587243 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 19:56:05.588039 (+ 796us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 19:56:05.588052 (+ 13us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.588106 (+ 54us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.588163 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20240501 19:56:05.588313 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.585017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.8.168.66:45073)
0501 19:56:05.585162 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.585167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.585185 (+ 18us) server_negotiation.cc:407] Connection header received
0501 19:56:05.585387 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.585391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.585457 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.585564 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:05.585878 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.586611 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.587208 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.587402 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.587461 (+ 59us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:05.587943 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:05.588051 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.588133 (+ 82us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.588181 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 19:56:05.592832 10761 raft_consensus.cc:1261] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b [term 1 FOLLOWER]: Refusing update from remote peer 5e7f6611c0e84cdab59c4063b64ab205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.592836 11041 raft_consensus.cc:1261] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4 [term 1 FOLLOWER]: Refusing update from remote peer 5e7f6611c0e84cdab59c4063b64ab205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.593429 11134 consensus_queue.cc:1035] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.593608 11118 consensus_queue.cc:1035] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.596320 11116 tablet_replica.cc:439] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b: TxnStatusTablet state changed. Reason: New leader 5e7f6611c0e84cdab59c4063b64ab205. Latest consensus state: current_term: 1 leader_uuid: "5e7f6611c0e84cdab59c4063b64ab205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } }
I20240501 19:56:05.596426 11116 tablet_replica.cc:442] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:05.597126 11117 tablet_replica.cc:439] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4: TxnStatusTablet state changed. Reason: New leader 5e7f6611c0e84cdab59c4063b64ab205. Latest consensus state: current_term: 1 leader_uuid: "5e7f6611c0e84cdab59c4063b64ab205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } }
I20240501 19:56:05.597237 11117 tablet_replica.cc:442] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:05.600270 11116 tablet_replica.cc:439] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e7f6611c0e84cdab59c4063b64ab205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } }
I20240501 19:56:05.600376 11116 tablet_replica.cc:442] T 42cb8c07a9414030af87e3263d8c90e6 P d38e614bf7c24d979ba5385e5074ba8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:05.600752 11117 tablet_replica.cc:439] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e7f6611c0e84cdab59c4063b64ab205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } }
I20240501 19:56:05.600854 11117 tablet_replica.cc:442] T 42cb8c07a9414030af87e3263d8c90e6 P a0e01564d00b4073a181140b862fd7f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:05.601855 11118 tablet_replica.cc:439] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e7f6611c0e84cdab59c4063b64ab205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } }
I20240501 19:56:05.601967 11118 tablet_replica.cc:442] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205: This TxnStatusTablet replica's current role is: LEADER
I20240501 19:56:05.602545 11141 mvcc.cc:205] Tried to move back new op lower bound from 7022974425462050816 to 7022974425373794304. Current Snapshot: MvccSnapshot[applied={T|T < 7022974425462050816}]
I20240501 19:56:05.602764 11118 tablet_replica.cc:439] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e7f6611c0e84cdab59c4063b64ab205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } } }
I20240501 19:56:05.602864 11118 tablet_replica.cc:442] T 42cb8c07a9414030af87e3263d8c90e6 P 5e7f6611c0e84cdab59c4063b64ab205: This TxnStatusTablet replica's current role is: LEADER
I20240501 19:56:05.611267 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.608341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45428 (local address 127.8.168.65:42745)
0501 19:56:05.608447 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.608450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.608464 (+ 14us) server_negotiation.cc:407] Connection header received
0501 19:56:05.608552 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.608554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.608605 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.608784 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:05.609208 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.609669 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.610287 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.610452 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.610505 (+ 53us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:05.610893 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:05.610991 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.611078 (+ 87us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.611126 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240501 19:56:05.621419 11151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.618359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:45073 (local address 127.8.168.65:44849)
0501 19:56:05.618605 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.618617 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.618705 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.619010 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.619013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.619021 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 19:56:05.619288 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.619295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.620262 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.620265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.620665 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.620673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.620813 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.620848 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 19:56:05.621204 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 19:56:05.621208 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.621251 (+ 43us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.621301 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":77,"threads_started":1}
I20240501 19:56:05.621441 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.618467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:44849 (local address 127.8.168.66:45073)
0501 19:56:05.618588 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.618593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.618678 (+ 85us) server_negotiation.cc:407] Connection header received
0501 19:56:05.618829 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.618832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.618884 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.618964 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:05.619395 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.620111 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.620812 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.621000 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.621047 (+ 47us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:05.621121 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:05.621190 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.621276 (+ 86us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.621317 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 19:56:05.628064 10901 raft_consensus.cc:1261] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 FOLLOWER]: Refusing update from remote peer d38e614bf7c24d979ba5385e5074ba8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.628494 11041 raft_consensus.cc:1261] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4 [term 1 FOLLOWER]: Refusing update from remote peer d38e614bf7c24d979ba5385e5074ba8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.628597 11121 consensus_queue.cc:1035] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.629087 11121 consensus_queue.cc:1035] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.634639 11140 mvcc.cc:205] Tried to move back new op lower bound from 7022974425608048640 to 7022974424558702592. Current Snapshot: MvccSnapshot[applied={T|T < 7022974425608048640}]
I20240501 19:56:05.636000 11139 mvcc.cc:205] Tried to move back new op lower bound from 7022974425608048640 to 7022974424558702592. Current Snapshot: MvccSnapshot[applied={T|T < 7022974425608048640}]
I20240501 19:56:05.652349 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.649620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:44459 (local address 127.8.168.65:42745)
0501 19:56:05.649781 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.649785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.649970 (+ 185us) server_negotiation.cc:407] Connection header received
0501 19:56:05.650083 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.650086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.650139 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.650207 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:05.650640 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.651106 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.651731 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.651905 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.651945 (+ 40us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:05.652006 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:05.652064 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.652168 (+ 104us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.652218 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240501 19:56:05.652632 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.649547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:42745 (local address 127.8.168.66:44459)
0501 19:56:05.649879 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.649894 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.649996 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.650214 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.650218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.650226 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 19:56:05.650534 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.650544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.651199 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.651203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.651629 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.651636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.651741 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.651785 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 19:56:05.652084 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 19:56:05.652091 (+ 7us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.652153 (+ 62us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.652218 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":124,"threads_started":1}
I20240501 19:56:05.678912 10604 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46756:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:05.679103 10604 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46756:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:05.681852 10604 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2: Sending DeleteTablet for 3 replicas of tablet ee6b06647ffe49b1a19a7921f219b20a
I20240501 19:56:05.682597 10741 tablet_service.cc:1514] Processing DeleteTablet for tablet ee6b06647ffe49b1a19a7921f219b20a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:05 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:45424
I20240501 19:56:05.683033 10881 tablet_service.cc:1514] Processing DeleteTablet for tablet ee6b06647ffe49b1a19a7921f219b20a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:05 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:47942
I20240501 19:56:05.683125 11021 tablet_service.cc:1514] Processing DeleteTablet for tablet ee6b06647ffe49b1a19a7921f219b20a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:05 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:58966
I20240501 19:56:05.683804 11159 tablet_replica.cc:330] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b: stopping tablet replica
I20240501 19:56:05.684121 11159 raft_consensus.cc:2229] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:05.684327 10604 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:56:05.684785 10604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:05.684933 11159 raft_consensus.cc:2258] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:05.686406 11159 ts_tablet_manager.cc:1875] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:05.689395 11159 ts_tablet_manager.cc:1888] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 19:56:05.689554 11159 log.cc:1196] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/wal/wals/ee6b06647ffe49b1a19a7921f219b20a
I20240501 19:56:05.689831 11021 tablet_service.cc:1467] Processing CreateTablet for tablet 299831791f0948df88c9c13f9d8a0a68 (DEFAULT_TABLE table=test-table [id=30e92c3e7efc4fff906f81746482023f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.690023 11159 ts_tablet_manager.cc:1909] T ee6b06647ffe49b1a19a7921f219b20a P d38e614bf7c24d979ba5385e5074ba8b: Deleting consensus metadata
I20240501 19:56:05.690115 11021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 299831791f0948df88c9c13f9d8a0a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.692181 11111 tablet_bootstrap.cc:492] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4: Bootstrap starting.
I20240501 19:56:05.692811 10881 tablet_service.cc:1467] Processing CreateTablet for tablet 299831791f0948df88c9c13f9d8a0a68 (DEFAULT_TABLE table=test-table [id=30e92c3e7efc4fff906f81746482023f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.692875 11161 tablet_replica.cc:330] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4: stopping tablet replica
I20240501 19:56:05.693056 10881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 299831791f0948df88c9c13f9d8a0a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.693127 11161 raft_consensus.cc:2229] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:05.693172 11111 tablet_bootstrap.cc:654] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.693343 11161 raft_consensus.cc:2258] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:05.694483 11111 tablet_bootstrap.cc:492] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4: No bootstrap required, opened a new log
I20240501 19:56:05.694563 11111 ts_tablet_manager.cc:1367] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:05.694891 11160 tablet_replica.cc:330] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205: stopping tablet replica
I20240501 19:56:05.695015 11111 raft_consensus.cc:348] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.695109 11111 raft_consensus.cc:374] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.695117 11160 raft_consensus.cc:2229] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:05.695138 11111 raft_consensus.cc:729] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e01564d00b4073a181140b862fd7f4, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.695184 10588 catalog_manager.cc:4830] TS d38e614bf7c24d979ba5385e5074ba8b (127.8.168.65:42745): tablet ee6b06647ffe49b1a19a7921f219b20a (table test-table [id=a88cdec672a44b1aa48c6b7e93a8e4d5]) successfully deleted
I20240501 19:56:05.695274 11160 raft_consensus.cc:2258] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:05.695250 11111 consensus_queue.cc:260] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.695439 11161 ts_tablet_manager.cc:1875] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:05.695489 11111 ts_tablet_manager.cc:1398] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:05.695678 10741 tablet_service.cc:1467] Processing CreateTablet for tablet 299831791f0948df88c9c13f9d8a0a68 (DEFAULT_TABLE table=test-table [id=30e92c3e7efc4fff906f81746482023f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.695916 10741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 299831791f0948df88c9c13f9d8a0a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.697908 11161 ts_tablet_manager.cc:1888] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 19:56:05.697952 11110 tablet_bootstrap.cc:492] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b: Bootstrap starting.
I20240501 19:56:05.697990 11161 log.cc:1196] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/ee6b06647ffe49b1a19a7921f219b20a
I20240501 19:56:05.698293 11161 ts_tablet_manager.cc:1909] T ee6b06647ffe49b1a19a7921f219b20a P a0e01564d00b4073a181140b862fd7f4: Deleting consensus metadata
I20240501 19:56:05.698618 11112 tablet_bootstrap.cc:492] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205: Bootstrap starting.
I20240501 19:56:05.698871 11110 tablet_bootstrap.cc:654] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.699386 10589 catalog_manager.cc:4830] TS a0e01564d00b4073a181140b862fd7f4 (127.8.168.67:44535): tablet ee6b06647ffe49b1a19a7921f219b20a (table test-table [id=a88cdec672a44b1aa48c6b7e93a8e4d5]) successfully deleted
I20240501 19:56:05.699440 11112 tablet_bootstrap.cc:654] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.700217 11110 tablet_bootstrap.cc:492] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b: No bootstrap required, opened a new log
I20240501 19:56:05.700299 11110 ts_tablet_manager.cc:1367] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:05.700675 11160 ts_tablet_manager.cc:1875] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:05.700690 11110 raft_consensus.cc:348] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.700824 11110 raft_consensus.cc:374] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.700853 11110 raft_consensus.cc:729] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38e614bf7c24d979ba5385e5074ba8b, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.700965 11110 consensus_queue.cc:260] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.701224 11110 ts_tablet_manager.cc:1398] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:05.703301 11160 ts_tablet_manager.cc:1888] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 19:56:05.703382 11160 log.cc:1196] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/ee6b06647ffe49b1a19a7921f219b20a
I20240501 19:56:05.703675 11160 ts_tablet_manager.cc:1909] T ee6b06647ffe49b1a19a7921f219b20a P 5e7f6611c0e84cdab59c4063b64ab205: Deleting consensus metadata
I20240501 19:56:05.704810 10591 catalog_manager.cc:4830] TS 5e7f6611c0e84cdab59c4063b64ab205 (127.8.168.66:45073): tablet ee6b06647ffe49b1a19a7921f219b20a (table test-table [id=a88cdec672a44b1aa48c6b7e93a8e4d5]) successfully deleted
I20240501 19:56:05.705449 11112 tablet_bootstrap.cc:492] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205: No bootstrap required, opened a new log
I20240501 19:56:05.705543 11112 ts_tablet_manager.cc:1367] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:05.705971 11112 raft_consensus.cc:348] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.706084 11112 raft_consensus.cc:374] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.706120 11112 raft_consensus.cc:729] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7f6611c0e84cdab59c4063b64ab205, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.706243 11112 consensus_queue.cc:260] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.706483 11112 ts_tablet_manager.cc:1398] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:05.847906 11117 raft_consensus.cc:482] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:05.848045 11117 raft_consensus.cc:504] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.849092 11117 leader_election.cc:288] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d38e614bf7c24d979ba5385e5074ba8b (127.8.168.65:42745), 5e7f6611c0e84cdab59c4063b64ab205 (127.8.168.66:45073)
I20240501 19:56:05.853312 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.849428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:56077 (local address 127.8.168.66:45073)
0501 19:56:05.849558 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.849562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.849780 (+ 218us) server_negotiation.cc:407] Connection header received
0501 19:56:05.849935 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.849939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.849992 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.850040 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:05.850565 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.851408 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.852411 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.852884 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.852926 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.853147 (+ 221us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.853209 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240501 19:56:05.853328 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.849363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:45073 (local address 127.8.168.67:56077)
0501 19:56:05.849661 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.849675 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.849795 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.850160 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.850162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.850177 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:05.850442 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.850450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.851512 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.851515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.852137 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.852148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.852990 (+ 842us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.853009 (+ 19us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.853110 (+ 101us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.853178 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":67,"threads_started":1}
I20240501 19:56:05.853790 10901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "299831791f0948df88c9c13f9d8a0a68" candidate_uuid: "a0e01564d00b4073a181140b862fd7f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7f6611c0e84cdab59c4063b64ab205" is_pre_election: true
I20240501 19:56:05.853935 10901 raft_consensus.cc:2454] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e01564d00b4073a181140b862fd7f4 in term 0.
I20240501 19:56:05.854290 10975 leader_election.cc:302] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7f6611c0e84cdab59c4063b64ab205, a0e01564d00b4073a181140b862fd7f4; no voters:
I20240501 19:56:05.854426 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.849373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:42745 (local address 127.8.168.67:49197)
0501 19:56:05.849889 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.849901 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.849975 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.850240 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.850243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.850256 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:05.850457 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.850463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.852472 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.852477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.853425 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.853435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.854124 (+ 689us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.854141 (+ 17us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.854210 (+ 69us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.854263 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":69,"threads_started":1}
I20240501 19:56:05.854444 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.849746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:49197 (local address 127.8.168.65:42745)
0501 19:56:05.849905 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.849910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.849946 (+ 36us) server_negotiation.cc:407] Connection header received
0501 19:56:05.850081 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.850085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.850139 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.850228 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:05.851574 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.852316 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.853570 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.854077 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.854118 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.854245 (+ 127us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.854302 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20240501 19:56:05.854627 11117 raft_consensus.cc:2781] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:05.854679 11117 raft_consensus.cc:482] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:05.854715 11117 raft_consensus.cc:3037] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.854840 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "299831791f0948df88c9c13f9d8a0a68" candidate_uuid: "a0e01564d00b4073a181140b862fd7f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38e614bf7c24d979ba5385e5074ba8b" is_pre_election: true
I20240501 19:56:05.854974 10761 raft_consensus.cc:2454] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e01564d00b4073a181140b862fd7f4 in term 0.
I20240501 19:56:05.855726 11117 raft_consensus.cc:504] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.856042 11117 leader_election.cc:288] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [CANDIDATE]: Term 1 election: Requested vote from peers d38e614bf7c24d979ba5385e5074ba8b (127.8.168.65:42745), 5e7f6611c0e84cdab59c4063b64ab205 (127.8.168.66:45073)
I20240501 19:56:05.856369 10901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "299831791f0948df88c9c13f9d8a0a68" candidate_uuid: "a0e01564d00b4073a181140b862fd7f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e7f6611c0e84cdab59c4063b64ab205"
I20240501 19:56:05.856443 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "299831791f0948df88c9c13f9d8a0a68" candidate_uuid: "a0e01564d00b4073a181140b862fd7f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38e614bf7c24d979ba5385e5074ba8b"
I20240501 19:56:05.856482 10901 raft_consensus.cc:3037] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.856539 10761 raft_consensus.cc:3037] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.857501 10761 raft_consensus.cc:2454] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e01564d00b4073a181140b862fd7f4 in term 1.
I20240501 19:56:05.857501 10901 raft_consensus.cc:2454] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e01564d00b4073a181140b862fd7f4 in term 1.
I20240501 19:56:05.857795 10975 leader_election.cc:302] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7f6611c0e84cdab59c4063b64ab205, a0e01564d00b4073a181140b862fd7f4; no voters:
I20240501 19:56:05.857972 11117 raft_consensus.cc:2781] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.858188 11117 raft_consensus.cc:686] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [term 1 LEADER]: Becoming Leader. State: Replica: a0e01564d00b4073a181140b862fd7f4, State: Running, Role: LEADER
I20240501 19:56:05.858440 11117 consensus_queue.cc:237] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:05.860792 10603 catalog_manager.cc:5484] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0e01564d00b4073a181140b862fd7f4 (127.8.168.67). New cstate: current_term: 1 leader_uuid: "a0e01564d00b4073a181140b862fd7f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:05.900974 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.898310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58982 (local address 127.8.168.67:44535)
0501 19:56:05.898414 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.898417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.898487 (+ 70us) server_negotiation.cc:407] Connection header received
0501 19:56:05.898626 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.898628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.898677 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.898769 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:05.899019 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.899471 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.900056 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.900175 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.900222 (+ 47us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:05.900644 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:05.900756 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.900830 (+ 74us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.900871 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240501 19:56:05.909797 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.906926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:45073 (local address 127.8.168.67:43007)
0501 19:56:05.907176 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.907190 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.907298 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.907521 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.907524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.907530 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 19:56:05.907975 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.907982 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.908619 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.908622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.909070 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.909078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.909189 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.909229 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 19:56:05.909529 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 19:56:05.909536 (+ 7us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.909590 (+ 54us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.909653 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":81,"threads_started":1}
I20240501 19:56:05.909788 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.907037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:43007 (local address 127.8.168.66:45073)
0501 19:56:05.907156 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.907159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.907248 (+ 89us) server_negotiation.cc:407] Connection header received
0501 19:56:05.907394 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.907397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.907438 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.907499 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:05.908082 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.908519 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.909172 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.909349 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.909389 (+ 40us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:05.909451 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:05.909511 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.909608 (+ 97us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.909650 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240501 19:56:05.915522 10901 raft_consensus.cc:1261] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 FOLLOWER]: Refusing update from remote peer a0e01564d00b4073a181140b862fd7f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.915832 10761 raft_consensus.cc:1261] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b [term 1 FOLLOWER]: Refusing update from remote peer a0e01564d00b4073a181140b862fd7f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.915940 11167 consensus_queue.cc:1035] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.916164 11117 consensus_queue.cc:1035] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.931527 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.929215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:44535 (local address 127.8.168.66:53649)
0501 19:56:05.929338 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:05.929353 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:05.929429 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:05.929632 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:05.929635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:05.929642 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 19:56:05.929822 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.929828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.930451 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.930455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:05.930850 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:05.930857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.930948 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.930976 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 19:56:05.931286 (+ 310us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 19:56:05.931291 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 19:56:05.931346 (+ 55us) client_negotiation.cc:240] Negotiation successful
0501 19:56:05.931415 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20240501 19:56:05.931527 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:05.929289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:53649 (local address 127.8.168.67:44535)
0501 19:56:05.929403 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:05.929407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:05.929419 (+ 12us) server_negotiation.cc:407] Connection header received
0501 19:56:05.929514 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:05.929517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:05.929560 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:05.929617 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:05.929913 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.930361 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:05.930944 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:05.931104 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:05.931141 (+ 37us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:05.931199 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:05.931258 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:05.931365 (+ 107us) server_negotiation.cc:299] Negotiation successful
0501 19:56:05.931403 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20240501 19:56:05.932299 11140 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20240501 19:56:07.487417 10603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46756:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:07.487586 10603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46756:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:07.490299 10603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2: Sending DeleteTablet for 3 replicas of tablet 299831791f0948df88c9c13f9d8a0a68
I20240501 19:56:07.490924 10881 tablet_service.cc:1514] Processing DeleteTablet for tablet 299831791f0948df88c9c13f9d8a0a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:07 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:47942
I20240501 19:56:07.490922 10741 tablet_service.cc:1514] Processing DeleteTablet for tablet 299831791f0948df88c9c13f9d8a0a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:07 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:45424
I20240501 19:56:07.491290 11200 tablet_replica.cc:330] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b: stopping tablet replica
I20240501 19:56:07.491290 11021 tablet_service.cc:1514] Processing DeleteTablet for tablet 299831791f0948df88c9c13f9d8a0a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:07 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:58966
I20240501 19:56:07.491494 11200 raft_consensus.cc:2229] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:07.491636 11199 tablet_replica.cc:330] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205: stopping tablet replica
I20240501 19:56:07.491648 11200 raft_consensus.cc:2258] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:07.491799 11199 raft_consensus.cc:2229] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:07.491956 11199 raft_consensus.cc:2258] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:07.492334 11200 ts_tablet_manager.cc:1875] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:07.492856 11199 ts_tablet_manager.cc:1875] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:07.494839 11200 ts_tablet_manager.cc:1888] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240501 19:56:07.494906 11200 log.cc:1196] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/wal/wals/299831791f0948df88c9c13f9d8a0a68
I20240501 19:56:07.495141 11200 ts_tablet_manager.cc:1909] T 299831791f0948df88c9c13f9d8a0a68 P d38e614bf7c24d979ba5385e5074ba8b: Deleting consensus metadata
I20240501 19:56:07.495534 11199 ts_tablet_manager.cc:1888] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240501 19:56:07.495615 11199 log.cc:1196] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/299831791f0948df88c9c13f9d8a0a68
I20240501 19:56:07.495779 11201 tablet_replica.cc:330] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4: stopping tablet replica
I20240501 19:56:07.495762 10588 catalog_manager.cc:4830] TS d38e614bf7c24d979ba5385e5074ba8b (127.8.168.65:42745): tablet 299831791f0948df88c9c13f9d8a0a68 (table test-table [id=30e92c3e7efc4fff906f81746482023f]) successfully deleted
I20240501 19:56:07.495906 11201 raft_consensus.cc:2229] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:07.496182 11201 raft_consensus.cc:2258] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:07.496194 11199 ts_tablet_manager.cc:1909] T 299831791f0948df88c9c13f9d8a0a68 P 5e7f6611c0e84cdab59c4063b64ab205: Deleting consensus metadata
I20240501 19:56:07.496766 11201 ts_tablet_manager.cc:1875] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:07.496922 10591 catalog_manager.cc:4830] TS 5e7f6611c0e84cdab59c4063b64ab205 (127.8.168.66:45073): tablet 299831791f0948df88c9c13f9d8a0a68 (table test-table [id=30e92c3e7efc4fff906f81746482023f]) successfully deleted
I20240501 19:56:07.499218 11201 ts_tablet_manager.cc:1888] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240501 19:56:07.499284 11201 log.cc:1196] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/299831791f0948df88c9c13f9d8a0a68
I20240501 19:56:07.499493 11201 ts_tablet_manager.cc:1909] T 299831791f0948df88c9c13f9d8a0a68 P a0e01564d00b4073a181140b862fd7f4: Deleting consensus metadata
I20240501 19:56:07.500092 10589 catalog_manager.cc:4830] TS a0e01564d00b4073a181140b862fd7f4 (127.8.168.67:44535): tablet 299831791f0948df88c9c13f9d8a0a68 (table test-table [id=30e92c3e7efc4fff906f81746482023f]) successfully deleted
May 01 19:56:07 dist-test-slave-tzct krb5kdc[10559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593367, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 01 19:56:07 dist-test-slave-tzct krb5kdc[10559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593367, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:07.517159 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:07.509651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.8.168.126:41585)
0501 19:56:07.509925 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:07.509929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:07.509944 (+ 15us) server_negotiation.cc:407] Connection header received
0501 19:56:07.510077 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:07.510080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:07.510124 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:07.510205 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:07.510993 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:07.511519 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:07.512128 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:07.512332 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:07.514366 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:07.514383 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:07.514385 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:07.514408 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:07.515637 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:07.516134 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:07.516137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:07.516138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:07.516182 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:07.516385 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:07.516388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:07.516390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:07.516532 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:07.516647 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:07.516883 (+ 236us) server_negotiation.cc:299] Negotiation successful
0501 19:56:07.517000 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
I20240501 19:56:07.519482 10603 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:56:07.519920 10603 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:07.524200 10741 tablet_service.cc:1467] Processing CreateTablet for tablet 22236e8372894992b752c5e8458ed9fc (DEFAULT_TABLE table=test-table [id=1c0969422d2440dc888a02232ec85ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.524386 10881 tablet_service.cc:1467] Processing CreateTablet for tablet 22236e8372894992b752c5e8458ed9fc (DEFAULT_TABLE table=test-table [id=1c0969422d2440dc888a02232ec85ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.524502 10741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22236e8372894992b752c5e8458ed9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.524441 11021 tablet_service.cc:1467] Processing CreateTablet for tablet 22236e8372894992b752c5e8458ed9fc (DEFAULT_TABLE table=test-table [id=1c0969422d2440dc888a02232ec85ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.524631 10881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22236e8372894992b752c5e8458ed9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.524711 11021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22236e8372894992b752c5e8458ed9fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.526486 11210 tablet_bootstrap.cc:492] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4: Bootstrap starting.
I20240501 19:56:07.526825 11211 tablet_bootstrap.cc:492] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205: Bootstrap starting.
I20240501 19:56:07.527343 11210 tablet_bootstrap.cc:654] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.527655 11211 tablet_bootstrap.cc:654] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.528291 11212 tablet_bootstrap.cc:492] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b: Bootstrap starting.
I20240501 19:56:07.528592 11210 tablet_bootstrap.cc:492] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4: No bootstrap required, opened a new log
I20240501 19:56:07.528682 11210 ts_tablet_manager.cc:1367] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:07.529158 11212 tablet_bootstrap.cc:654] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.529136 11210 raft_consensus.cc:348] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.529232 11210 raft_consensus.cc:374] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.529278 11210 raft_consensus.cc:729] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e01564d00b4073a181140b862fd7f4, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.529414 11210 consensus_queue.cc:260] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.529683 11210 ts_tablet_manager.cc:1398] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:07.529964 11211 tablet_bootstrap.cc:492] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205: No bootstrap required, opened a new log
I20240501 19:56:07.530042 11211 ts_tablet_manager.cc:1367] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:07.530378 11211 raft_consensus.cc:348] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.530463 11211 raft_consensus.cc:374] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.530490 11211 raft_consensus.cc:729] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7f6611c0e84cdab59c4063b64ab205, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.530612 11211 consensus_queue.cc:260] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.530815 11211 ts_tablet_manager.cc:1398] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:07.531711 11212 tablet_bootstrap.cc:492] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b: No bootstrap required, opened a new log
I20240501 19:56:07.531845 11212 ts_tablet_manager.cc:1367] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:07.532480 11212 raft_consensus.cc:348] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.532644 11212 raft_consensus.cc:374] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.532687 11212 raft_consensus.cc:729] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38e614bf7c24d979ba5385e5074ba8b, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.532848 11212 consensus_queue.cc:260] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.533239 11212 ts_tablet_manager.cc:1398] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:07.566956 11194 raft_consensus.cc:482] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:07.567081 11194 raft_consensus.cc:504] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.567485 11194 leader_election.cc:288] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d38e614bf7c24d979ba5385e5074ba8b (127.8.168.65:42745), a0e01564d00b4073a181140b862fd7f4 (127.8.168.67:44535)
I20240501 19:56:07.567838 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22236e8372894992b752c5e8458ed9fc" candidate_uuid: "5e7f6611c0e84cdab59c4063b64ab205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38e614bf7c24d979ba5385e5074ba8b" is_pre_election: true
I20240501 19:56:07.567854 11041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22236e8372894992b752c5e8458ed9fc" candidate_uuid: "5e7f6611c0e84cdab59c4063b64ab205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e01564d00b4073a181140b862fd7f4" is_pre_election: true
I20240501 19:56:07.567972 11041 raft_consensus.cc:2454] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7f6611c0e84cdab59c4063b64ab205 in term 0.
I20240501 19:56:07.567976 10761 raft_consensus.cc:2454] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7f6611c0e84cdab59c4063b64ab205 in term 0.
I20240501 19:56:07.568289 10833 leader_election.cc:302] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7f6611c0e84cdab59c4063b64ab205, a0e01564d00b4073a181140b862fd7f4; no voters:
I20240501 19:56:07.568459 11194 raft_consensus.cc:2781] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:07.568518 11194 raft_consensus.cc:482] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:07.568543 11194 raft_consensus.cc:3037] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.569454 11194 raft_consensus.cc:504] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.569756 11194 leader_election.cc:288] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [CANDIDATE]: Term 1 election: Requested vote from peers d38e614bf7c24d979ba5385e5074ba8b (127.8.168.65:42745), a0e01564d00b4073a181140b862fd7f4 (127.8.168.67:44535)
I20240501 19:56:07.570093 11041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22236e8372894992b752c5e8458ed9fc" candidate_uuid: "5e7f6611c0e84cdab59c4063b64ab205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e01564d00b4073a181140b862fd7f4"
I20240501 19:56:07.570133 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22236e8372894992b752c5e8458ed9fc" candidate_uuid: "5e7f6611c0e84cdab59c4063b64ab205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38e614bf7c24d979ba5385e5074ba8b"
I20240501 19:56:07.570219 11041 raft_consensus.cc:3037] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.570226 10761 raft_consensus.cc:3037] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.571182 11041 raft_consensus.cc:2454] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e7f6611c0e84cdab59c4063b64ab205 in term 1.
I20240501 19:56:07.571182 10761 raft_consensus.cc:2454] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e7f6611c0e84cdab59c4063b64ab205 in term 1.
I20240501 19:56:07.571458 10833 leader_election.cc:302] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7f6611c0e84cdab59c4063b64ab205, a0e01564d00b4073a181140b862fd7f4; no voters:
I20240501 19:56:07.571607 11194 raft_consensus.cc:2781] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:07.571689 11194 raft_consensus.cc:686] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 LEADER]: Becoming Leader. State: Replica: 5e7f6611c0e84cdab59c4063b64ab205, State: Running, Role: LEADER
I20240501 19:56:07.571830 11194 consensus_queue.cc:237] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.573452 10602 catalog_manager.cc:5484] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e7f6611c0e84cdab59c4063b64ab205 (127.8.168.66). New cstate: current_term: 1 leader_uuid: "5e7f6611c0e84cdab59c4063b64ab205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:07.586071 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:07.583494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47966 (local address 127.8.168.66:45073)
0501 19:56:07.583712 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:07.583716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:07.583726 (+ 10us) server_negotiation.cc:407] Connection header received
0501 19:56:07.583761 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:07.583763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:07.583805 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:07.583883 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:07.584236 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:07.584773 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:07.585462 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:07.585629 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:07.585669 (+ 40us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:07.585735 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:07.585813 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:07.585913 (+ 100us) server_negotiation.cc:299] Negotiation successful
0501 19:56:07.585952 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20240501 19:56:07.593453 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:07.591120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:51717 (local address 127.8.168.66:45073)
0501 19:56:07.591209 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:07.591212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:07.591276 (+ 64us) server_negotiation.cc:407] Connection header received
0501 19:56:07.591374 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:07.591376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:07.591415 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:07.591471 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:07.591775 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:07.592259 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:07.592905 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:07.593034 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:07.593084 (+ 50us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:07.593142 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:07.593192 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:07.593281 (+ 89us) server_negotiation.cc:299] Negotiation successful
0501 19:56:07.593319 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240501 19:56:07.593472 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:07.590980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:45073 (local address 127.8.168.66:51717)
0501 19:56:07.591211 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:07.591223 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:07.591305 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:07.591483 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:07.591485 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:07.591490 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 19:56:07.591671 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:07.591675 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:07.592346 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:07.592348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:07.592826 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:07.592834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:07.592921 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:07.592942 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 19:56:07.593207 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 19:56:07.593213 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 19:56:07.593261 (+ 48us) client_negotiation.cc:240] Negotiation successful
0501 19:56:07.593319 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":77,"threads_started":1}
I20240501 19:56:07.598546 11041 raft_consensus.cc:1261] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4 [term 1 FOLLOWER]: Refusing update from remote peer 5e7f6611c0e84cdab59c4063b64ab205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:07.598624 10761 raft_consensus.cc:1261] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b [term 1 FOLLOWER]: Refusing update from remote peer 5e7f6611c0e84cdab59c4063b64ab205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:07.598910 11180 consensus_queue.cc:1035] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:07.599037 11118 consensus_queue.cc:1035] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:07.629662 10602 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46766:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:07.629810 10602 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46766:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:07.631763 10602 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2: Sending DeleteTablet for 3 replicas of tablet 22236e8372894992b752c5e8458ed9fc
I20240501 19:56:07.632326 10881 tablet_service.cc:1514] Processing DeleteTablet for tablet 22236e8372894992b752c5e8458ed9fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:07 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:47942
I20240501 19:56:07.632340 10741 tablet_service.cc:1514] Processing DeleteTablet for tablet 22236e8372894992b752c5e8458ed9fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:07 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:45424
I20240501 19:56:07.632550 11199 tablet_replica.cc:330] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205: stopping tablet replica
I20240501 19:56:07.632561 11200 tablet_replica.cc:330] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b: stopping tablet replica
I20240501 19:56:07.632552 11021 tablet_service.cc:1514] Processing DeleteTablet for tablet 22236e8372894992b752c5e8458ed9fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:07 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:58966
I20240501 19:56:07.632659 11199 raft_consensus.cc:2229] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:07.632689 11200 raft_consensus.cc:2229] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:07.632789 11201 tablet_replica.cc:330] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4: stopping tablet replica
I20240501 19:56:07.632867 11200 raft_consensus.cc:2258] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:07.632882 11201 raft_consensus.cc:2229] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:07.632994 11201 raft_consensus.cc:2258] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:07.632992 11199 raft_consensus.cc:2258] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:07.633661 11200 ts_tablet_manager.cc:1875] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:07.633667 11199 ts_tablet_manager.cc:1875] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:07.633906 11201 ts_tablet_manager.cc:1875] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:07.635979 11199 ts_tablet_manager.cc:1888] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 19:56:07.635979 11201 ts_tablet_manager.cc:1888] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 19:56:07.635986 11200 ts_tablet_manager.cc:1888] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 19:56:07.636044 11199 log.cc:1196] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/22236e8372894992b752c5e8458ed9fc
I20240501 19:56:07.636044 11201 log.cc:1196] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/22236e8372894992b752c5e8458ed9fc
I20240501 19:56:07.636049 11200 log.cc:1196] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-0/wal/wals/22236e8372894992b752c5e8458ed9fc
I20240501 19:56:07.636281 11201 ts_tablet_manager.cc:1909] T 22236e8372894992b752c5e8458ed9fc P a0e01564d00b4073a181140b862fd7f4: Deleting consensus metadata
I20240501 19:56:07.636281 11199 ts_tablet_manager.cc:1909] T 22236e8372894992b752c5e8458ed9fc P 5e7f6611c0e84cdab59c4063b64ab205: Deleting consensus metadata
I20240501 19:56:07.636301 11200 ts_tablet_manager.cc:1909] T 22236e8372894992b752c5e8458ed9fc P d38e614bf7c24d979ba5385e5074ba8b: Deleting consensus metadata
I20240501 19:56:07.637167 10588 catalog_manager.cc:4830] TS d38e614bf7c24d979ba5385e5074ba8b (127.8.168.65:42745): tablet 22236e8372894992b752c5e8458ed9fc (table test-table [id=1c0969422d2440dc888a02232ec85ee6]) successfully deleted
I20240501 19:56:07.637621 10602 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:56:07.637699 10589 catalog_manager.cc:4830] TS a0e01564d00b4073a181140b862fd7f4 (127.8.168.67:44535): tablet 22236e8372894992b752c5e8458ed9fc (table test-table [id=1c0969422d2440dc888a02232ec85ee6]) successfully deleted
I20240501 19:56:07.637679 10591 catalog_manager.cc:4830] TS 5e7f6611c0e84cdab59c4063b64ab205 (127.8.168.66:45073): tablet 22236e8372894992b752c5e8458ed9fc (table test-table [id=1c0969422d2440dc888a02232ec85ee6]) successfully deleted
W20240501 19:56:07.638171 10602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:07.642391 10741 tablet_service.cc:1467] Processing CreateTablet for tablet a7ae953637d54414b99536838a9c9047 (DEFAULT_TABLE table=test-table [id=6603ca5b4cae4e5ebbd5a31f19dfa325]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.642560 11021 tablet_service.cc:1467] Processing CreateTablet for tablet a7ae953637d54414b99536838a9c9047 (DEFAULT_TABLE table=test-table [id=6603ca5b4cae4e5ebbd5a31f19dfa325]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.642652 10741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ae953637d54414b99536838a9c9047. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.642794 11021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ae953637d54414b99536838a9c9047. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.644433 11210 tablet_bootstrap.cc:492] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4: Bootstrap starting.
I20240501 19:56:07.644620 10881 tablet_service.cc:1467] Processing CreateTablet for tablet a7ae953637d54414b99536838a9c9047 (DEFAULT_TABLE table=test-table [id=6603ca5b4cae4e5ebbd5a31f19dfa325]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.644683 11212 tablet_bootstrap.cc:492] T a7ae953637d54414b99536838a9c9047 P d38e614bf7c24d979ba5385e5074ba8b: Bootstrap starting.
I20240501 19:56:07.644877 10881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ae953637d54414b99536838a9c9047. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.645272 11210 tablet_bootstrap.cc:654] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.645502 11212 tablet_bootstrap.cc:654] T a7ae953637d54414b99536838a9c9047 P d38e614bf7c24d979ba5385e5074ba8b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.646557 11212 tablet_bootstrap.cc:492] T a7ae953637d54414b99536838a9c9047 P d38e614bf7c24d979ba5385e5074ba8b: No bootstrap required, opened a new log
I20240501 19:56:07.646641 11212 ts_tablet_manager.cc:1367] T a7ae953637d54414b99536838a9c9047 P d38e614bf7c24d979ba5385e5074ba8b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:07.646781 11211 tablet_bootstrap.cc:492] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205: Bootstrap starting.
I20240501 19:56:07.647023 11212 raft_consensus.cc:348] T a7ae953637d54414b99536838a9c9047 P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.647117 11212 raft_consensus.cc:374] T a7ae953637d54414b99536838a9c9047 P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.647145 11212 raft_consensus.cc:729] T a7ae953637d54414b99536838a9c9047 P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38e614bf7c24d979ba5385e5074ba8b, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.647254 11212 consensus_queue.cc:260] T a7ae953637d54414b99536838a9c9047 P d38e614bf7c24d979ba5385e5074ba8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.647497 11212 ts_tablet_manager.cc:1398] T a7ae953637d54414b99536838a9c9047 P d38e614bf7c24d979ba5385e5074ba8b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:07.647558 11211 tablet_bootstrap.cc:654] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.648263 11210 tablet_bootstrap.cc:492] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4: No bootstrap required, opened a new log
I20240501 19:56:07.648347 11210 ts_tablet_manager.cc:1367] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 19:56:07.648823 11211 tablet_bootstrap.cc:492] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205: No bootstrap required, opened a new log
I20240501 19:56:07.648779 11210 raft_consensus.cc:348] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.648874 11210 raft_consensus.cc:374] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.648891 11211 ts_tablet_manager.cc:1367] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:07.648902 11210 raft_consensus.cc:729] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e01564d00b4073a181140b862fd7f4, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.649020 11210 consensus_queue.cc:260] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.649276 11210 ts_tablet_manager.cc:1398] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:07.649255 11211 raft_consensus.cc:348] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.649341 11211 raft_consensus.cc:374] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.649370 11211 raft_consensus.cc:729] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7f6611c0e84cdab59c4063b64ab205, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.649482 11211 consensus_queue.cc:260] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.649716 11211 ts_tablet_manager.cc:1398] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:07.881853 11180 raft_consensus.cc:482] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:07.881987 11180 raft_consensus.cc:504] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.882381 11180 leader_election.cc:288] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d38e614bf7c24d979ba5385e5074ba8b (127.8.168.65:42745), a0e01564d00b4073a181140b862fd7f4 (127.8.168.67:44535)
I20240501 19:56:07.882889 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ae953637d54414b99536838a9c9047" candidate_uuid: "5e7f6611c0e84cdab59c4063b64ab205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38e614bf7c24d979ba5385e5074ba8b" is_pre_election: true
I20240501 19:56:07.882877 11041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ae953637d54414b99536838a9c9047" candidate_uuid: "5e7f6611c0e84cdab59c4063b64ab205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e01564d00b4073a181140b862fd7f4" is_pre_election: true
I20240501 19:56:07.883050 11041 raft_consensus.cc:2454] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7f6611c0e84cdab59c4063b64ab205 in term 0.
I20240501 19:56:07.883050 10761 raft_consensus.cc:2454] T a7ae953637d54414b99536838a9c9047 P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7f6611c0e84cdab59c4063b64ab205 in term 0.
I20240501 19:56:07.883404 10832 leader_election.cc:302] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7f6611c0e84cdab59c4063b64ab205, d38e614bf7c24d979ba5385e5074ba8b; no voters:
I20240501 19:56:07.883603 11180 raft_consensus.cc:2781] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:07.883662 11180 raft_consensus.cc:482] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:07.883692 11180 raft_consensus.cc:3037] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.884703 11180 raft_consensus.cc:504] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.885037 11180 leader_election.cc:288] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [CANDIDATE]: Term 1 election: Requested vote from peers d38e614bf7c24d979ba5385e5074ba8b (127.8.168.65:42745), a0e01564d00b4073a181140b862fd7f4 (127.8.168.67:44535)
I20240501 19:56:07.885376 11041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ae953637d54414b99536838a9c9047" candidate_uuid: "5e7f6611c0e84cdab59c4063b64ab205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e01564d00b4073a181140b862fd7f4"
I20240501 19:56:07.885509 11041 raft_consensus.cc:3037] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.885478 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ae953637d54414b99536838a9c9047" candidate_uuid: "5e7f6611c0e84cdab59c4063b64ab205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38e614bf7c24d979ba5385e5074ba8b"
I20240501 19:56:07.885586 10761 raft_consensus.cc:3037] T a7ae953637d54414b99536838a9c9047 P d38e614bf7c24d979ba5385e5074ba8b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.886544 11041 raft_consensus.cc:2454] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e7f6611c0e84cdab59c4063b64ab205 in term 1.
I20240501 19:56:07.886544 10761 raft_consensus.cc:2454] T a7ae953637d54414b99536838a9c9047 P d38e614bf7c24d979ba5385e5074ba8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e7f6611c0e84cdab59c4063b64ab205 in term 1.
I20240501 19:56:07.886822 10833 leader_election.cc:302] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e7f6611c0e84cdab59c4063b64ab205, a0e01564d00b4073a181140b862fd7f4; no voters:
I20240501 19:56:07.886981 11180 raft_consensus.cc:2781] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:07.887063 11180 raft_consensus.cc:686] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 LEADER]: Becoming Leader. State: Replica: 5e7f6611c0e84cdab59c4063b64ab205, State: Running, Role: LEADER
I20240501 19:56:07.887197 11180 consensus_queue.cc:237] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } }
I20240501 19:56:07.888769 10603 catalog_manager.cc:5484] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e7f6611c0e84cdab59c4063b64ab205 (127.8.168.66). New cstate: current_term: 1 leader_uuid: "5e7f6611c0e84cdab59c4063b64ab205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e7f6611c0e84cdab59c4063b64ab205" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:07.948103 10761 raft_consensus.cc:1261] T a7ae953637d54414b99536838a9c9047 P d38e614bf7c24d979ba5385e5074ba8b [term 1 FOLLOWER]: Refusing update from remote peer 5e7f6611c0e84cdab59c4063b64ab205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:07.948150 11041 raft_consensus.cc:1261] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4 [term 1 FOLLOWER]: Refusing update from remote peer 5e7f6611c0e84cdab59c4063b64ab205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:07.948462 11118 consensus_queue.cc:1035] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d38e614bf7c24d979ba5385e5074ba8b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:07.948606 11194 consensus_queue.cc:1035] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e01564d00b4073a181140b862fd7f4" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:56:07.960084 11178 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20240501 19:56:09.505380 10603 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46766:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:09.505532 10603 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46766:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:09.507822 10603 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 328441c5b38e489cad2d0a34c4129ae2: Sending DeleteTablet for 3 replicas of tablet a7ae953637d54414b99536838a9c9047
I20240501 19:56:09.508292 10741 tablet_service.cc:1514] Processing DeleteTablet for tablet a7ae953637d54414b99536838a9c9047 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:09 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:45424
I20240501 19:56:09.508531 11021 tablet_service.cc:1514] Processing DeleteTablet for tablet a7ae953637d54414b99536838a9c9047 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:09 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:58966
I20240501 19:56:09.508622 10880 tablet_service.cc:1514] Processing DeleteTablet for tablet a7ae953637d54414b99536838a9c9047 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:09 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:47942
I20240501 19:56:09.508865 11251 tablet_replica.cc:330] T a7ae953637d54414b99536838a9c9047 P d38e614bf7c24d979ba5385e5074ba8b: stopping tablet replica
I20240501 19:56:09.508941 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 10676
I20240501 19:56:09.508980 11251 raft_consensus.cc:2229] T a7ae953637d54414b99536838a9c9047 P d38e614bf7c24d979ba5385e5074ba8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:09.509778 11253 tablet_replica.cc:330] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205: stopping tablet replica
I20240501 19:56:09.509877 11253 raft_consensus.cc:2229] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:09.510064 11253 raft_consensus.cc:2258] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:09.510563 11253 ts_tablet_manager.cc:1875] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:09.511050 11252 tablet_replica.cc:330] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4: stopping tablet replica
I20240501 19:56:09.511169 11252 raft_consensus.cc:2229] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:09.511274 11252 raft_consensus.cc:2258] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:09.511819 11252 ts_tablet_manager.cc:1875] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:09.512820 11253 ts_tablet_manager.cc:1888] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240501 19:56:09.512889 11253 log.cc:1196] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/a7ae953637d54414b99536838a9c9047
I20240501 19:56:09.513145 11253 ts_tablet_manager.cc:1909] T a7ae953637d54414b99536838a9c9047 P 5e7f6611c0e84cdab59c4063b64ab205: Deleting consensus metadata
I20240501 19:56:09.513732 10591 catalog_manager.cc:4830] TS 5e7f6611c0e84cdab59c4063b64ab205 (127.8.168.66:45073): tablet a7ae953637d54414b99536838a9c9047 (table test-table [id=6603ca5b4cae4e5ebbd5a31f19dfa325]) successfully deleted
I20240501 19:56:09.514160 11252 ts_tablet_manager.cc:1888] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240501 19:56:09.514235 11252 log.cc:1196] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/a7ae953637d54414b99536838a9c9047
I20240501 19:56:09.514485 11252 ts_tablet_manager.cc:1909] T a7ae953637d54414b99536838a9c9047 P a0e01564d00b4073a181140b862fd7f4: Deleting consensus metadata
I20240501 19:56:09.515096 10589 catalog_manager.cc:4830] TS a0e01564d00b4073a181140b862fd7f4 (127.8.168.67:44535): tablet a7ae953637d54414b99536838a9c9047 (table test-table [id=6603ca5b4cae4e5ebbd5a31f19dfa325]) successfully deleted
W20240501 19:56:09.516414 10588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.168.65:42745 (error 108)
I20240501 19:56:09.516603 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 10816
I20240501 19:56:09.517973 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:09.517612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:42745 (local address 127.0.0.1:45434)
0501 19:56:09.517840 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:09.517892 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:42745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":73,"threads_started":1}
W20240501 19:56:09.518092 10588 catalog_manager.cc:4573] TS d38e614bf7c24d979ba5385e5074ba8b (127.8.168.65:42745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7ae953637d54414b99536838a9c9047: Network error: Client connection negotiation failed: client connection to 127.8.168.65:42745: connect: Connection refused (error 111)
I20240501 19:56:09.522503 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 10956
I20240501 19:56:09.527971 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 10572
2024-05-01T19:56:09Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6193 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 19:56:09 dist-test-slave-tzct krb5kdc[11258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 19:56:09 dist-test-slave-tzct krb5kdc[11258](info): set up 2 sockets
May 01 19:56:09 dist-test-slave-tzct krb5kdc[11258](info): commencing operation
krb5kdc: starting...
W20240501 19:56:10.428864 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.880s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 19:56:10 dist-test-slave-tzct krb5kdc[11258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T19:56:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 19:56:10.540606 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:40773
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:40773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:10.646173 11271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:10.646438 11271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:10.646514 11271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:10.649643 11271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:10.649708 11271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:10.649745 11271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:10.649770 11271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:10.649799 11271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:10.654175 11271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:40773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:40773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:10.655093 11271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:10.655968 11271 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:10.663312 11276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:10.663902 11271 server_base.cc:1034] running on GCE node
W20240501 19:56:10.664376 11279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:10.664801 11277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:10.665342 11271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:10.666299 11271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:10.667477 11271 hybrid_clock.cc:648] HybridClock initialized: now 1714593370667463 us; error 26 us; skew 500 ppm
May 01 19:56:10 dist-test-slave-tzct krb5kdc[11258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593370, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:10.670121 11271 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 19:56:10.671074 11271 webserver.cc:460] Webserver started at http://127.8.168.126:46483/ using document root <none> and password file <none>
I20240501 19:56:10.671581 11271 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:10.671633 11271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:10.671820 11271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:10.673622 11271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "31d2f34f4d314d07b36e7fd97385cabd"
format_stamp: "Formatted at 2024-05-01 19:56:10 on dist-test-slave-tzct"
server_key: "660295c3b7b1ad8a47795526da25947f"
server_key_iv: "ac9f7520c000b009c87e1248fe87137c"
server_key_version: "encryptionkey@0"
I20240501 19:56:10.674099 11271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "31d2f34f4d314d07b36e7fd97385cabd"
format_stamp: "Formatted at 2024-05-01 19:56:10 on dist-test-slave-tzct"
server_key: "660295c3b7b1ad8a47795526da25947f"
server_key_iv: "ac9f7520c000b009c87e1248fe87137c"
server_key_version: "encryptionkey@0"
I20240501 19:56:10.677311 11271 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:10.679397 11286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:10.680359 11271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:10.680465 11271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "31d2f34f4d314d07b36e7fd97385cabd"
format_stamp: "Formatted at 2024-05-01 19:56:10 on dist-test-slave-tzct"
server_key: "660295c3b7b1ad8a47795526da25947f"
server_key_iv: "ac9f7520c000b009c87e1248fe87137c"
server_key_version: "encryptionkey@0"
I20240501 19:56:10.680547 11271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:10.714751 11271 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:10.717777 11271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:10.717949 11271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:10.724426 11271 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:40773
I20240501 19:56:10.724445 11338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:40773 every 8 connection(s)
I20240501 19:56:10.725402 11271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:56:10.726341 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 11271
I20240501 19:56:10.726456 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:56:10.726712 8865 external_mini_cluster.cc:1428] Setting key 4c28bfe99d9b87a06d537f0cf00fbe55
I20240501 19:56:10.728595 11339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
May 01 19:56:10 dist-test-slave-tzct krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:10.734328 11339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd: Bootstrap starting.
I20240501 19:56:10.736304 11339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:10.737114 11339 log.cc:824] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:10.738756 11342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:10.727879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51498 (local address 127.8.168.126:40773)
0501 19:56:10.728240 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:10.728249 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:10.728282 (+ 33us) server_negotiation.cc:407] Connection header received
0501 19:56:10.728931 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:10.728950 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:10.729227 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:10.729515 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:10.730477 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:10.731276 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:10.732105 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:10.732305 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:10.734690 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:10.734732 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:10.734743 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:10.734778 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:10.736411 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:10.736878 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:10.736883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:10.736887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:10.736961 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:10.737178 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:10.737180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:10.737181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:10.737533 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:10.737719 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:10.737975 (+ 256us) server_negotiation.cc:299] Negotiation successful
0501 19:56:10.738217 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
I20240501 19:56:10.738883 11339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd: No bootstrap required, opened a new log
I20240501 19:56:10.741277 11339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d2f34f4d314d07b36e7fd97385cabd" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40773 } }
I20240501 19:56:10.741452 11339 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:10.741508 11339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d2f34f4d314d07b36e7fd97385cabd, State: Initialized, Role: FOLLOWER
I20240501 19:56:10.741874 11339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d2f34f4d314d07b36e7fd97385cabd" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40773 } }
I20240501 19:56:10.741981 11339 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:10.742038 11339 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:10.742128 11339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:10.742900 11339 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d2f34f4d314d07b36e7fd97385cabd" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40773 } }
I20240501 19:56:10.743207 11339 leader_election.cc:302] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31d2f34f4d314d07b36e7fd97385cabd; no voters:
I20240501 19:56:10.743475 11339 leader_election.cc:288] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:10.743595 11344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:10.743772 11344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [term 1 LEADER]: Becoming Leader. State: Replica: 31d2f34f4d314d07b36e7fd97385cabd, State: Running, Role: LEADER
I20240501 19:56:10.744115 11344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d2f34f4d314d07b36e7fd97385cabd" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40773 } }
I20240501 19:56:10.744444 11339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:10.745635 11345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31d2f34f4d314d07b36e7fd97385cabd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d2f34f4d314d07b36e7fd97385cabd" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40773 } } }
I20240501 19:56:10.745668 11346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31d2f34f4d314d07b36e7fd97385cabd. Latest consensus state: current_term: 1 leader_uuid: "31d2f34f4d314d07b36e7fd97385cabd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d2f34f4d314d07b36e7fd97385cabd" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40773 } } }
I20240501 19:56:10.745743 11345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:10.745756 11346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:10.746199 11354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:10.749073 11354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:10.753814 11354 catalog_manager.cc:1347] Generated new cluster ID: 888433b007c9487a84874a6367ba4d7b
I20240501 19:56:10.753895 11354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:10.777782 11354 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:10.778678 11354 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:10.783669 11354 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 31d2f34f4d314d07b36e7fd97385cabd: Generated new TSK 0
I20240501 19:56:10.784397 11354 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
I20240501 19:56:10.818993 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:40773
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:10.920626 11365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:10.920850 11365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:10.920908 11365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:10.923861 11365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:10.923913 11365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:10.923995 11365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:56:10.927920 11365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:40773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:10.928714 11365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:10.929533 11365 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:10.936975 11370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:10.937554 11373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:10.937984 11371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:10.938654 11365 server_base.cc:1034] running on GCE node
I20240501 19:56:10.938993 11365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:10.939477 11365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:10.940624 11365 hybrid_clock.cc:648] HybridClock initialized: now 1714593370940586 us; error 53 us; skew 500 ppm
May 01 19:56:10 dist-test-slave-tzct krb5kdc[11258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593370, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:10.943166 11365 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 19:56:10.944013 11365 webserver.cc:460] Webserver started at http://127.8.168.65:38823/ using document root <none> and password file <none>
I20240501 19:56:10.944504 11365 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:10.944597 11365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:10.944845 11365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:10.946630 11365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "626e9ca902af48c7bbc8697a8ac7a42c"
format_stamp: "Formatted at 2024-05-01 19:56:10 on dist-test-slave-tzct"
server_key: "f1833156761c8ddd45525e4784b5efcd"
server_key_iv: "86be27687876c218786803a89a47eec6"
server_key_version: "encryptionkey@0"
I20240501 19:56:10.947084 11365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "626e9ca902af48c7bbc8697a8ac7a42c"
format_stamp: "Formatted at 2024-05-01 19:56:10 on dist-test-slave-tzct"
server_key: "f1833156761c8ddd45525e4784b5efcd"
server_key_iv: "86be27687876c218786803a89a47eec6"
server_key_version: "encryptionkey@0"
I20240501 19:56:10.950178 11365 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:10.952160 11380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:10.953136 11365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:10.953238 11365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "626e9ca902af48c7bbc8697a8ac7a42c"
format_stamp: "Formatted at 2024-05-01 19:56:10 on dist-test-slave-tzct"
server_key: "f1833156761c8ddd45525e4784b5efcd"
server_key_iv: "86be27687876c218786803a89a47eec6"
server_key_version: "encryptionkey@0"
I20240501 19:56:10.953315 11365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:10.973536 11365 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:10.976050 11365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:10.976203 11365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:10.976706 11365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:10.977470 11365 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:10.977523 11365 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:10.977574 11365 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:10.977600 11365 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:10.985563 11365 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:44397
I20240501 19:56:10.985610 11493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:44397 every 8 connection(s)
I20240501 19:56:10.986410 11365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 19:56:10.994088 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 11365
I20240501 19:56:10.994186 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
May 01 19:56:10 dist-test-slave-tzct krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593370, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:10.994421 8865 external_mini_cluster.cc:1428] Setting key dba91b7c5c36a7f76f78746dae9fc5e7
I20240501 19:56:10.998353 11342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:10.987945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:60513 (local address 127.8.168.126:40773)
0501 19:56:10.988065 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:10.988069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:10.988676 (+ 607us) server_negotiation.cc:407] Connection header received
0501 19:56:10.989494 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:10.989497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:10.989567 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:10.989638 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:10.991225 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:10.991723 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:10.992495 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:10.992696 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:10.994868 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:10.994887 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:10.994889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:10.994915 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:10.996472 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:10.996973 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:10.996976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:10.996978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:10.997030 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:10.997263 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:10.997266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:10.997268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:10.997440 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:10.997523 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:10.998124 (+ 601us) server_negotiation.cc:299] Negotiation successful
0501 19:56:10.998226 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240501 19:56:10.999084 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:10.988159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:40773 (local address 127.8.168.65:60513)
0501 19:56:10.988542 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:10.988575 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:10.989298 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:10.989808 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:10.989816 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:10.990159 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:10.991078 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:10.991091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:10.991920 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:10.991924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:10.992358 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:10.992368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:10.992601 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:10.993279 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:10.993304 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:10.994706 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:10.996594 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:10.996598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:10.996606 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:10.996888 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:10.997107 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:10.997109 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:10.997110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:10.997182 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:10.997561 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:10.997566 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:10.997893 (+ 327us) client_negotiation.cc:769] Sending connection context
0501 19:56:10.998100 (+ 207us) client_negotiation.cc:240] Negotiation successful
0501 19:56:10.998344 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":107,"threads_started":1}
I20240501 19:56:11.000154 11494 heartbeater.cc:347] Connected to a master server at 127.8.168.126:40773
I20240501 19:56:11.000376 11494 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:11.001013 11494 heartbeater.cc:511] Master 127.8.168.126:40773 requested a full tablet report, sending...
I20240501 19:56:11.002440 11303 ts_manager.cc:194] Registered new tserver with Master: 626e9ca902af48c7bbc8697a8ac7a42c (127.8.168.65:44397)
WARNING: no policy specified for kudu/127.8.168.66@KRBTEST.COM; defaulting to no policy
I20240501 19:56:11.003540 11303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:60513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
I20240501 19:56:11.017989 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:40773
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:11.119948 11499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:11.120174 11499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:11.120257 11499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:11.123250 11499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:11.123320 11499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:11.123423 11499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:56:11.127427 11499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:40773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:11.128264 11499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:11.129129 11499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:11.137313 11507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:11.137614 11504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:11.137867 11505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:11.138413 11499 server_base.cc:1034] running on GCE node
I20240501 19:56:11.138770 11499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:11.139335 11499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:11.140488 11499 hybrid_clock.cc:648] HybridClock initialized: now 1714593371140476 us; error 28 us; skew 500 ppm
May 01 19:56:11 dist-test-slave-tzct krb5kdc[11258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593371, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:11.143078 11499 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 19:56:11.144035 11499 webserver.cc:460] Webserver started at http://127.8.168.66:43145/ using document root <none> and password file <none>
I20240501 19:56:11.144553 11499 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:11.144615 11499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:11.144884 11499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:11.146795 11499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "50b3e0d1a55447fe81125864005e6711"
format_stamp: "Formatted at 2024-05-01 19:56:11 on dist-test-slave-tzct"
server_key: "22bdf8e30d738ac43a66746f3f1bf468"
server_key_iv: "0a0ddd8d0c8028ace934aab500e4d640"
server_key_version: "encryptionkey@0"
I20240501 19:56:11.147284 11499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "50b3e0d1a55447fe81125864005e6711"
format_stamp: "Formatted at 2024-05-01 19:56:11 on dist-test-slave-tzct"
server_key: "22bdf8e30d738ac43a66746f3f1bf468"
server_key_iv: "0a0ddd8d0c8028ace934aab500e4d640"
server_key_version: "encryptionkey@0"
I20240501 19:56:11.150729 11499 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240501 19:56:11.152935 11514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:11.153971 11499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:11.154079 11499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "50b3e0d1a55447fe81125864005e6711"
format_stamp: "Formatted at 2024-05-01 19:56:11 on dist-test-slave-tzct"
server_key: "22bdf8e30d738ac43a66746f3f1bf468"
server_key_iv: "0a0ddd8d0c8028ace934aab500e4d640"
server_key_version: "encryptionkey@0"
I20240501 19:56:11.154161 11499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:11.176663 11499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:11.179566 11499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:11.179723 11499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:11.180243 11499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:11.181051 11499 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:11.181107 11499 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:11.181160 11499 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:11.181183 11499 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:11.189781 11499 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:37659
I20240501 19:56:11.189867 11627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:37659 every 8 connection(s)
I20240501 19:56:11.190660 11499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 19:56:11.192960 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 11499
I20240501 19:56:11.193089 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 19:56:11.193368 8865 external_mini_cluster.cc:1428] Setting key 0897d2c92759a0ee104c5e451531de42
May 01 19:56:11 dist-test-slave-tzct krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593371, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.168.67@KRBTEST.COM; defaulting to no policy
I20240501 19:56:11.202751 11342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:11.192293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:58795 (local address 127.8.168.126:40773)
0501 19:56:11.192419 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:11.192423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:11.193173 (+ 750us) server_negotiation.cc:407] Connection header received
0501 19:56:11.194146 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:11.194149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:11.194201 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:11.194278 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:11.196022 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:11.196501 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:11.197148 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:11.197326 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:11.199322 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:11.199338 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:11.199341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:11.199364 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:11.200663 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:11.201324 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:11.201329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:11.201330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:11.201382 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:11.201653 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:11.201656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:11.201658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:11.201846 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:11.201951 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:11.202445 (+ 494us) server_negotiation.cc:299] Negotiation successful
0501 19:56:11.202574 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.67@KRBTEST.COM" created.
I20240501 19:56:11.203408 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:11.192576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:40773 (local address 127.8.168.66:58795)
0501 19:56:11.193030 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:11.193082 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:11.193946 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:11.194472 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:11.194482 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:11.194981 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:11.195881 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:11.195894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:11.196608 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:11.196611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:11.197037 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:11.197046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:11.197258 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:11.197830 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:11.197856 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:11.199174 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:11.200780 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:11.200786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:11.200796 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:11.201205 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:11.201473 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:11.201476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:11.201479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:11.201563 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:11.201964 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:11.201970 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:11.202206 (+ 236us) client_negotiation.cc:769] Sending connection context
0501 19:56:11.202391 (+ 185us) client_negotiation.cc:240] Negotiation successful
0501 19:56:11.202645 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":124,"threads_started":1}
I20240501 19:56:11.204469 11628 heartbeater.cc:347] Connected to a master server at 127.8.168.126:40773
I20240501 19:56:11.204684 11628 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:11.205197 11628 heartbeater.cc:511] Master 127.8.168.126:40773 requested a full tablet report, sending...
I20240501 19:56:11.207571 11303 ts_manager.cc:194] Registered new tserver with Master: 50b3e0d1a55447fe81125864005e6711 (127.8.168.66:37659)
I20240501 19:56:11.208920 11303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:58795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
I20240501 19:56:11.217463 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:40773
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:11.317852 11633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:11.318063 11633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:11.318116 11633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:11.321094 11633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:11.321151 11633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:11.321223 11633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:56:11.325150 11633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:40773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:11.325920 11633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:11.326676 11633 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:11.334069 11639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:11.334547 11641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:11.335083 11638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:11.335578 11633 server_base.cc:1034] running on GCE node
I20240501 19:56:11.335916 11633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:11.336426 11633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:11.337575 11633 hybrid_clock.cc:648] HybridClock initialized: now 1714593371337554 us; error 34 us; skew 500 ppm
May 01 19:56:11 dist-test-slave-tzct krb5kdc[11258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593371, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:11.339931 11633 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 19:56:11.340797 11633 webserver.cc:460] Webserver started at http://127.8.168.67:35391/ using document root <none> and password file <none>
I20240501 19:56:11.341308 11633 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:11.341365 11633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:11.341527 11633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:11.343246 11633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "cc9ea5027f81400e9e14b3eb56351706"
format_stamp: "Formatted at 2024-05-01 19:56:11 on dist-test-slave-tzct"
server_key: "3ad94a91ef2f7c6af2fefe0233e25f27"
server_key_iv: "49ffc0c9cc7ba73893e0786026c298f9"
server_key_version: "encryptionkey@0"
I20240501 19:56:11.343698 11633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "cc9ea5027f81400e9e14b3eb56351706"
format_stamp: "Formatted at 2024-05-01 19:56:11 on dist-test-slave-tzct"
server_key: "3ad94a91ef2f7c6af2fefe0233e25f27"
server_key_iv: "49ffc0c9cc7ba73893e0786026c298f9"
server_key_version: "encryptionkey@0"
I20240501 19:56:11.346972 11633 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 19:56:11.349113 11648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:11.350076 11633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:11.350179 11633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "cc9ea5027f81400e9e14b3eb56351706"
format_stamp: "Formatted at 2024-05-01 19:56:11 on dist-test-slave-tzct"
server_key: "3ad94a91ef2f7c6af2fefe0233e25f27"
server_key_iv: "49ffc0c9cc7ba73893e0786026c298f9"
server_key_version: "encryptionkey@0"
I20240501 19:56:11.350270 11633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:11.359542 11633 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:11.362352 11633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:11.362509 11633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:11.363045 11633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:11.363790 11633 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:11.363843 11633 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:11.363893 11633 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:11.363919 11633 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:11.373116 11633 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:33687
I20240501 19:56:11.373148 11761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:33687 every 8 connection(s)
I20240501 19:56:11.373999 11633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
May 01 19:56:11 dist-test-slave-tzct krb5kdc[11258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593371, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:11.381770 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 11633
I20240501 19:56:11.381876 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 19:56:11.382094 8865 external_mini_cluster.cc:1428] Setting key 10f360bbc5055640d8d4d42819c8750d
I20240501 19:56:11.385650 11342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:11.375604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:38283 (local address 127.8.168.126:40773)
0501 19:56:11.375772 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:11.375776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:11.376368 (+ 592us) server_negotiation.cc:407] Connection header received
0501 19:56:11.377197 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:11.377201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:11.377274 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:11.377345 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:11.378899 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:11.379393 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:11.380014 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:11.380194 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:11.382349 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:11.382371 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:11.382374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:11.382399 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:11.383903 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:11.384336 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:11.384339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:11.384340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:11.384397 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:11.384648 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:11.384652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:11.384654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:11.384863 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:11.384950 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:11.385424 (+ 474us) server_negotiation.cc:299] Negotiation successful
0501 19:56:11.385522 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20240501 19:56:11.386193 11764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:11.375843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:40773 (local address 127.8.168.67:38283)
0501 19:56:11.376225 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:11.376265 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:11.376990 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:11.377506 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:11.377514 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:11.377873 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:11.378754 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:11.378768 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:11.379509 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:11.379512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:11.379897 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:11.379905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:11.380109 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:11.380692 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:11.380749 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:11.382203 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:11.384008 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:11.384012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:11.384020 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:11.384235 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:11.384472 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:11.384474 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:11.384475 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:11.384545 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:11.384973 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:11.384978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:11.385186 (+ 208us) client_negotiation.cc:769] Sending connection context
0501 19:56:11.385353 (+ 167us) client_negotiation.cc:240] Negotiation successful
0501 19:56:11.385550 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":94,"threads_started":1}
I20240501 19:56:11.387145 11762 heartbeater.cc:347] Connected to a master server at 127.8.168.126:40773
I20240501 19:56:11.387347 11762 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:11.387782 11762 heartbeater.cc:511] Master 127.8.168.126:40773 requested a full tablet report, sending...
I20240501 19:56:11.388701 11303 ts_manager.cc:194] Registered new tserver with Master: cc9ea5027f81400e9e14b3eb56351706 (127.8.168.67:33687)
I20240501 19:56:11.389273 11303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:38283
I20240501 19:56:11.395483 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240501 19:56:11.406090 11342 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:11.404838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51510 (local address 127.8.168.126:40773)
0501 19:56:11.404984 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:11.404988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:11.405110 (+ 122us) server_negotiation.cc:407] Connection header received
0501 19:56:11.405236 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:11.405238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:11.405290 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:11.405359 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:11.405964 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51510: BlockingRecv error: recv got EOF from 127.0.0.1:51510 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20240501 19:56:11.407083 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 11365
I20240501 19:56:11.411686 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 11499
I20240501 19:56:11.415729 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 11633
I20240501 19:56:11.419991 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 11271
2024-05-01T19:56:11Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1890 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 19:56:11 dist-test-slave-tzct krb5kdc[11774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 19:56:11 dist-test-slave-tzct krb5kdc[11774](info): set up 2 sockets
May 01 19:56:11 dist-test-slave-tzct krb5kdc[11774](info): commencing operation
krb5kdc: starting...
W20240501 19:56:12.312956 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.873s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 19:56:12 dist-test-slave-tzct krb5kdc[11774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T19:56:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 19:56:12.424060 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:39959
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:33049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:39959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:12.525442 11787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:12.525681 11787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:12.525733 11787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:12.528766 11787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:12.528833 11787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:12.528856 11787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:12.528873 11787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:12.528892 11787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:12.533078 11787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:33049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:39959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:39959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:12.534075 11787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:12.534906 11787 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:56:12.542431 11787 server_base.cc:1034] running on GCE node
W20240501 19:56:12.542953 11795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:12.543275 11793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:12.543282 11792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:12.543821 11787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:12.544742 11787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:12.545892 11787 hybrid_clock.cc:648] HybridClock initialized: now 1714593372545867 us; error 38 us; skew 500 ppm
May 01 19:56:12 dist-test-slave-tzct krb5kdc[11774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593372, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:12.548509 11787 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 19:56:12.549377 11787 webserver.cc:460] Webserver started at http://127.8.168.126:44415/ using document root <none> and password file <none>
I20240501 19:56:12.549870 11787 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:12.549921 11787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:12.550081 11787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:12.551769 11787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "92ee38a02fbf40af907534f1568e358b"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-tzct"
server_key: "69b0fb9a5b1d291d86781a70271de80b"
server_key_iv: "eb0f81a1bb47663c895ea55b51f13602"
server_key_version: "encryptionkey@0"
I20240501 19:56:12.552215 11787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "92ee38a02fbf40af907534f1568e358b"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-tzct"
server_key: "69b0fb9a5b1d291d86781a70271de80b"
server_key_iv: "eb0f81a1bb47663c895ea55b51f13602"
server_key_version: "encryptionkey@0"
I20240501 19:56:12.555370 11787 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:12.557458 11802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:12.558506 11787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:12.558611 11787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "92ee38a02fbf40af907534f1568e358b"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-tzct"
server_key: "69b0fb9a5b1d291d86781a70271de80b"
server_key_iv: "eb0f81a1bb47663c895ea55b51f13602"
server_key_version: "encryptionkey@0"
I20240501 19:56:12.558692 11787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:12.571053 11787 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:12.573858 11787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:12.574033 11787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:12.580569 11787 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:39959
I20240501 19:56:12.580586 11854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:39959 every 8 connection(s)
I20240501 19:56:12.581499 11787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:56:12.584115 11855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:12.588752 11855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b: Bootstrap starting.
I20240501 19:56:12.588932 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 11787
I20240501 19:56:12.589047 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:56:12.589252 8865 external_mini_cluster.cc:1428] Setting key 439ad1b071370337ac52305a0d37c221
I20240501 19:56:12.591012 11855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:12.591655 11855 log.cc:824] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:12.593297 11855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b: No bootstrap required, opened a new log
May 01 19:56:12 dist-test-slave-tzct krb5kdc[11774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:12.596300 11855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ee38a02fbf40af907534f1568e358b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39959 } }
I20240501 19:56:12.596473 11855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:12.596527 11855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ee38a02fbf40af907534f1568e358b, State: Initialized, Role: FOLLOWER
I20240501 19:56:12.596943 11855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ee38a02fbf40af907534f1568e358b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39959 } }
I20240501 19:56:12.597112 11855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:12.597177 11855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:12.597280 11855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:12.598088 11855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ee38a02fbf40af907534f1568e358b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39959 } }
I20240501 19:56:12.598383 11855 leader_election.cc:302] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92ee38a02fbf40af907534f1568e358b; no voters:
I20240501 19:56:12.598618 11855 leader_election.cc:288] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:12.598692 11860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:12.598870 11860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [term 1 LEADER]: Becoming Leader. State: Replica: 92ee38a02fbf40af907534f1568e358b, State: Running, Role: LEADER
I20240501 19:56:12.599129 11860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ee38a02fbf40af907534f1568e358b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39959 } }
I20240501 19:56:12.599704 11855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:12.600209 11860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92ee38a02fbf40af907534f1568e358b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ee38a02fbf40af907534f1568e358b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39959 } } }
I20240501 19:56:12.600411 11860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:12.600646 11860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92ee38a02fbf40af907534f1568e358b. Latest consensus state: current_term: 1 leader_uuid: "92ee38a02fbf40af907534f1568e358b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ee38a02fbf40af907534f1568e358b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39959 } } }
I20240501 19:56:12.600857 11860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:12.601239 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:12.590264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.8.168.126:39959)
0501 19:56:12.590666 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:12.590674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:12.590706 (+ 32us) server_negotiation.cc:407] Connection header received
0501 19:56:12.591315 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:12.591332 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:12.591601 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:12.591923 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:12.592804 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:12.593673 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:12.594282 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:12.594543 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:12.596737 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:12.596760 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:12.596774 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:12.596814 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:12.598515 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:12.598919 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:12.598925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:12.598930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:12.599022 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:12.599232 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:12.599235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:12.599237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:12.599560 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:12.599754 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:12.600241 (+ 487us) server_negotiation.cc:299] Negotiation successful
0501 19:56:12.600495 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":138,"threads_started":1}
I20240501 19:56:12.601652 11866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:12.603991 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 19:56:12.604573 11866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:12.609270 11866 catalog_manager.cc:1347] Generated new cluster ID: c54a9588e85a45f596180a18579cb51f
I20240501 19:56:12.609400 11866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:12.620628 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:12.616162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42490 (local address 127.8.168.126:39959)
0501 19:56:12.616264 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:12.616268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:12.616524 (+ 256us) server_negotiation.cc:407] Connection header received
0501 19:56:12.616627 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:12.616630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:12.616685 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:12.616806 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:12.617475 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:12.618225 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:12.619006 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:12.619201 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:12.619671 (+ 470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:12.619691 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:12.619694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:56:12.620173 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:12.620206 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:56:12.620225 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:12.620314 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:12.620426 (+ 112us) server_negotiation.cc:299] Negotiation successful
0501 19:56:12.620490 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240501 19:56:12.621162 11866 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:12.622272 11866 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:12.634377 11866 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 92ee38a02fbf40af907534f1568e358b: Generated new TSK 0
I20240501 19:56:12.635043 11866 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:56:12.641480 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 11787
2024-05-01T19:56:12Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1219 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 19:56:12 dist-test-slave-tzct krb5kdc[11888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 19:56:12 dist-test-slave-tzct krb5kdc[11888](info): set up 2 sockets
May 01 19:56:12 dist-test-slave-tzct krb5kdc[11888](info): commencing operation
krb5kdc: starting...
W20240501 19:56:13.533052 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.874s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 19:56:13 dist-test-slave-tzct krb5kdc[11888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T19:56:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 19:56:13.639487 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:39855
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:42707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:39855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:13.739701 11901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:13.739953 11901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:13.740023 11901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:13.743036 11901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:13.743094 11901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:13.743124 11901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:13.743146 11901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:13.743175 11901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:13.747344 11901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:42707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:39855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:39855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:13.748214 11901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:13.749054 11901 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:13.756500 11906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:13.756592 11907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:13.757022 11909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:13.757678 11901 server_base.cc:1034] running on GCE node
I20240501 19:56:13.758208 11901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:13.759132 11901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:13.760283 11901 hybrid_clock.cc:648] HybridClock initialized: now 1714593373760266 us; error 32 us; skew 500 ppm
May 01 19:56:13 dist-test-slave-tzct krb5kdc[11888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593373, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:13.762980 11901 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 19:56:13.763870 11901 webserver.cc:460] Webserver started at http://127.8.168.126:44617/ using document root <none> and password file <none>
I20240501 19:56:13.764385 11901 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:13.764441 11901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:13.764611 11901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:13.766444 11901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "4f64fa024b04423da9d1bc0e1ccec65d"
format_stamp: "Formatted at 2024-05-01 19:56:13 on dist-test-slave-tzct"
server_key: "1df557c885d200785c016db474c4ed39"
server_key_iv: "87cbec5bb45b064b02ddc6d542025e8e"
server_key_version: "encryptionkey@0"
I20240501 19:56:13.766911 11901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "4f64fa024b04423da9d1bc0e1ccec65d"
format_stamp: "Formatted at 2024-05-01 19:56:13 on dist-test-slave-tzct"
server_key: "1df557c885d200785c016db474c4ed39"
server_key_iv: "87cbec5bb45b064b02ddc6d542025e8e"
server_key_version: "encryptionkey@0"
I20240501 19:56:13.770728 11901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240501 19:56:13.773075 11916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:13.774159 11901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:13.774295 11901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "4f64fa024b04423da9d1bc0e1ccec65d"
format_stamp: "Formatted at 2024-05-01 19:56:13 on dist-test-slave-tzct"
server_key: "1df557c885d200785c016db474c4ed39"
server_key_iv: "87cbec5bb45b064b02ddc6d542025e8e"
server_key_version: "encryptionkey@0"
I20240501 19:56:13.774384 11901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:13.795303 11901 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:13.798154 11901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:13.798300 11901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:13.805465 11901 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:39855
I20240501 19:56:13.805506 11968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:39855 every 8 connection(s)
I20240501 19:56:13.806717 11901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:56:13.809252 11969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:13.814019 11969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d: Bootstrap starting.
I20240501 19:56:13.814456 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 11901
I20240501 19:56:13.814579 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:56:13.814824 8865 external_mini_cluster.cc:1428] Setting key 37df7de2aff82a52762b479e5eeec713
I20240501 19:56:13.816462 11969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:13.817184 11969 log.cc:824] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:13.818964 11969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d: No bootstrap required, opened a new log
May 01 19:56:13 dist-test-slave-tzct krb5kdc[11888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:13.822191 11969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f64fa024b04423da9d1bc0e1ccec65d" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39855 } }
I20240501 19:56:13.822391 11969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:13.822450 11969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f64fa024b04423da9d1bc0e1ccec65d, State: Initialized, Role: FOLLOWER
I20240501 19:56:13.822916 11969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f64fa024b04423da9d1bc0e1ccec65d" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39855 } }
I20240501 19:56:13.823076 11969 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:13.823148 11969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:13.823268 11969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:13.826956 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:13.815996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39346 (local address 127.8.168.126:39855)
0501 19:56:13.816354 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:13.816363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:13.816397 (+ 34us) server_negotiation.cc:407] Connection header received
0501 19:56:13.817013 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:13.817031 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:13.817383 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:13.817726 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:13.818641 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:13.819454 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:13.820102 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:13.820362 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:13.822645 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:13.822668 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:13.822681 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:13.822724 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:13.824666 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:13.825117 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:13.825122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:13.825126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:13.825199 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:13.825387 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:13.825389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:13.825390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:13.825710 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:13.825922 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:13.826122 (+ 200us) server_negotiation.cc:299] Negotiation successful
0501 19:56:13.826332 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":111,"threads_started":1}
I20240501 19:56:13.843011 11969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f64fa024b04423da9d1bc0e1ccec65d" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39855 } }
I20240501 19:56:13.843490 11969 leader_election.cc:302] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f64fa024b04423da9d1bc0e1ccec65d; no voters:
I20240501 19:56:13.843801 11969 leader_election.cc:288] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:13.843917 11974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:13.844106 11974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [term 1 LEADER]: Becoming Leader. State: Replica: 4f64fa024b04423da9d1bc0e1ccec65d, State: Running, Role: LEADER
I20240501 19:56:13.844576 11974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f64fa024b04423da9d1bc0e1ccec65d" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39855 } }
I20240501 19:56:13.844930 11969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:13.846228 11975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f64fa024b04423da9d1bc0e1ccec65d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f64fa024b04423da9d1bc0e1ccec65d" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39855 } } }
I20240501 19:56:13.846278 11976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f64fa024b04423da9d1bc0e1ccec65d. Latest consensus state: current_term: 1 leader_uuid: "4f64fa024b04423da9d1bc0e1ccec65d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f64fa024b04423da9d1bc0e1ccec65d" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39855 } } }
I20240501 19:56:13.846366 11976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:13.846366 11975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:13.846840 11985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:13.849676 11985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:13.854665 11985 catalog_manager.cc:1347] Generated new cluster ID: 3a30d5f89b96425092fb5eaaff95cc5e
I20240501 19:56:13.854733 11985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:13.868139 11985 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:13.869123 11985 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:13.879276 11985 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4f64fa024b04423da9d1bc0e1ccec65d: Generated new TSK 0
I20240501 19:56:13.879932 11985 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
I20240501 19:56:13.905730 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:39855
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:42707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:14.008562 11995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:14.008819 11995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:14.008896 11995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:14.011912 11995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:14.011976 11995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:14.012071 11995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:56:14.015998 11995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:42707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:39855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:14.016892 11995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:14.017709 11995 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:14.025381 12000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:14.025931 11995 server_base.cc:1034] running on GCE node
W20240501 19:56:14.026367 12001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:14.026850 12003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:14.027204 11995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:14.027740 11995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:14.028880 11995 hybrid_clock.cc:648] HybridClock initialized: now 1714593374028870 us; error 25 us; skew 500 ppm
May 01 19:56:14 dist-test-slave-tzct krb5kdc[11888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593374, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:14.031368 11995 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 19:56:14.032233 11995 webserver.cc:460] Webserver started at http://127.8.168.65:39307/ using document root <none> and password file <none>
I20240501 19:56:14.032799 11995 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:14.032857 11995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:14.033037 11995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:14.034780 11995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "953febb72a0d4d02bdfc7c163e97b232"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-tzct"
server_key: "ba34533cdd4a1a5c791d4765e808c2df"
server_key_iv: "a29a4d143e306da4581483ca244f361c"
server_key_version: "encryptionkey@0"
I20240501 19:56:14.035233 11995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "953febb72a0d4d02bdfc7c163e97b232"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-tzct"
server_key: "ba34533cdd4a1a5c791d4765e808c2df"
server_key_iv: "a29a4d143e306da4581483ca244f361c"
server_key_version: "encryptionkey@0"
I20240501 19:56:14.038511 11995 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:14.040855 12010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.041872 11995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:14.042037 11995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "953febb72a0d4d02bdfc7c163e97b232"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-tzct"
server_key: "ba34533cdd4a1a5c791d4765e808c2df"
server_key_iv: "a29a4d143e306da4581483ca244f361c"
server_key_version: "encryptionkey@0"
I20240501 19:56:14.042145 11995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:14.069123 11995 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:14.072362 11995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:14.072571 11995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:14.073187 11995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:14.074321 11995 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:14.074399 11995 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.074460 11995 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:14.074488 11995 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.083202 11995 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:35751
I20240501 19:56:14.083228 12123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:35751 every 8 connection(s)
I20240501 19:56:14.084131 11995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 19:56:14.091171 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 11995
I20240501 19:56:14.091310 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 19:56:14.091567 8865 external_mini_cluster.cc:1428] Setting key 901e7916f760307653376d4fc222e8f5
May 01 19:56:14 dist-test-slave-tzct krb5kdc[11888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593374, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:14.096155 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:14.085832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:55999 (local address 127.8.168.126:39855)
0501 19:56:14.085967 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:14.085971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:14.086585 (+ 614us) server_negotiation.cc:407] Connection header received
0501 19:56:14.087782 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:14.087786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:14.087842 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:14.087916 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:14.089192 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:14.089704 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:14.090317 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:14.090505 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:14.092947 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:14.092963 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:14.092966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:14.092994 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:14.094397 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:14.094843 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:14.094846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:14.094848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:14.094892 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:14.095139 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:14.095142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:14.095144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:14.095287 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:14.095371 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:14.095838 (+ 467us) server_negotiation.cc:299] Negotiation successful
0501 19:56:14.095999 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240501 19:56:14.096765 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:14.086072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39855 (local address 127.8.168.65:55999)
0501 19:56:14.086453 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:14.086487 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:14.087562 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:14.088090 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:14.088098 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:14.088463 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:14.089054 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:14.089067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:14.089815 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:14.089818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:14.090210 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:14.090217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:14.090419 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:14.091068 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:14.091093 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:14.092784 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:14.094511 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:14.094516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:14.094527 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:14.094753 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:14.094970 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:14.094973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:14.094975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:14.095058 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:14.095381 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:14.095387 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:14.095614 (+ 227us) client_negotiation.cc:769] Sending connection context
0501 19:56:14.095790 (+ 176us) client_negotiation.cc:240] Negotiation successful
0501 19:56:14.096067 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":105,"threads_started":1}
I20240501 19:56:14.098031 12124 heartbeater.cc:347] Connected to a master server at 127.8.168.126:39855
I20240501 19:56:14.098241 12124 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:14.098726 12124 heartbeater.cc:511] Master 127.8.168.126:39855 requested a full tablet report, sending...
I20240501 19:56:14.100476 11933 ts_manager.cc:194] Registered new tserver with Master: 953febb72a0d4d02bdfc7c163e97b232 (127.8.168.65:35751)
WARNING: no policy specified for kudu/127.8.168.66@KRBTEST.COM; defaulting to no policy
I20240501 19:56:14.101788 11933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:55999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
I20240501 19:56:14.117193 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:39855
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:42707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:14.219647 12129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:14.219911 12129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:14.219993 12129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:14.223091 12129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:14.223165 12129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:14.223281 12129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:56:14.227290 12129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:42707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:39855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:14.228188 12129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:14.229055 12129 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:14.237504 12137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:14.237983 12135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:14.238533 12134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:14.238945 12129 server_base.cc:1034] running on GCE node
I20240501 19:56:14.239385 12129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:14.239969 12129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:14.241147 12129 hybrid_clock.cc:648] HybridClock initialized: now 1714593374241125 us; error 41 us; skew 500 ppm
May 01 19:56:14 dist-test-slave-tzct krb5kdc[11888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593374, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:14.244220 12129 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 19:56:14.245250 12129 webserver.cc:460] Webserver started at http://127.8.168.66:33813/ using document root <none> and password file <none>
I20240501 19:56:14.245870 12129 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:14.245944 12129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:14.246138 12129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:14.248108 12129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "84551c457a7b46679a4dfc0edf0388a9"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-tzct"
server_key: "6b50ccc700de20ae27e6aba1f3c6d590"
server_key_iv: "029f263787bbcd9e92203c0d78730cf5"
server_key_version: "encryptionkey@0"
I20240501 19:56:14.248576 12129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "84551c457a7b46679a4dfc0edf0388a9"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-tzct"
server_key: "6b50ccc700de20ae27e6aba1f3c6d590"
server_key_iv: "029f263787bbcd9e92203c0d78730cf5"
server_key_version: "encryptionkey@0"
I20240501 19:56:14.252010 12129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:14.254356 12144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.255357 12129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:14.255470 12129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "84551c457a7b46679a4dfc0edf0388a9"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-tzct"
server_key: "6b50ccc700de20ae27e6aba1f3c6d590"
server_key_iv: "029f263787bbcd9e92203c0d78730cf5"
server_key_version: "encryptionkey@0"
I20240501 19:56:14.255556 12129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:14.279552 12129 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:14.282524 12129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:14.282696 12129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:14.283247 12129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:14.284045 12129 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:14.284094 12129 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.284143 12129 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:14.284173 12129 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.292387 12129 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:37333
I20240501 19:56:14.292472 12257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:37333 every 8 connection(s)
I20240501 19:56:14.293352 12129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
May 01 19:56:14 dist-test-slave-tzct krb5kdc[11888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593374, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:14.302385 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 12129
I20240501 19:56:14.302515 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 19:56:14.302810 8865 external_mini_cluster.cc:1428] Setting key 417ae6ed2af40a840dcc818bd9ecffba
I20240501 19:56:14.305529 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:14.295045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:60101 (local address 127.8.168.126:39855)
0501 19:56:14.295218 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:14.295223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:14.295818 (+ 595us) server_negotiation.cc:407] Connection header received
0501 19:56:14.297096 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:14.297099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:14.297148 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:14.297221 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:14.298502 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:14.298986 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:14.299606 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:14.299728 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:14.302026 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:14.302045 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:14.302047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:14.302071 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:14.303505 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:14.304109 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:14.304113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:14.304115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:14.304173 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:14.304473 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:14.304477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:14.304479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:14.304629 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:14.304739 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:14.305237 (+ 498us) server_negotiation.cc:299] Negotiation successful
0501 19:56:14.305372 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20240501 19:56:14.306104 12260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:14.295281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39855 (local address 127.8.168.66:60101)
0501 19:56:14.295674 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:14.295708 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:14.296902 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:14.297412 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:14.297420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:14.297796 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:14.298378 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:14.298392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:14.299104 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:14.299107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:14.299514 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:14.299521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:14.299692 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:14.300239 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:14.300295 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:14.301885 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:14.303647 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:14.303652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:14.303662 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:14.303982 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:14.304262 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:14.304265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:14.304267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:14.304380 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:14.304751 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:14.304757 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:14.305007 (+ 250us) client_negotiation.cc:769] Sending connection context
0501 19:56:14.305188 (+ 181us) client_negotiation.cc:240] Negotiation successful
0501 19:56:14.305451 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":97,"threads_started":1}
I20240501 19:56:14.307066 12258 heartbeater.cc:347] Connected to a master server at 127.8.168.126:39855
I20240501 19:56:14.307276 12258 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:14.307770 12258 heartbeater.cc:511] Master 127.8.168.126:39855 requested a full tablet report, sending...
I20240501 19:56:14.308764 11933 ts_manager.cc:194] Registered new tserver with Master: 84551c457a7b46679a4dfc0edf0388a9 (127.8.168.66:37333)
I20240501 19:56:14.309294 11933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:60101
WARNING: no policy specified for kudu/127.8.168.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
I20240501 19:56:14.327556 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:39855
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:42707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:14.432902 12263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:14.433130 12263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:14.433192 12263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:14.436194 12263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:14.436251 12263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:14.436339 12263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:56:14.440259 12263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:42707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:39855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:14.441145 12263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:14.441984 12263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:14.449719 12271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:14.450126 12268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:14.450791 12269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:14.451216 12263 server_base.cc:1034] running on GCE node
I20240501 19:56:14.451558 12263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:14.452102 12263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:14.453239 12263 hybrid_clock.cc:648] HybridClock initialized: now 1714593374453226 us; error 30 us; skew 500 ppm
May 01 19:56:14 dist-test-slave-tzct krb5kdc[11888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593374, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:14.455796 12263 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 19:56:14.456686 12263 webserver.cc:460] Webserver started at http://127.8.168.67:42959/ using document root <none> and password file <none>
I20240501 19:56:14.457218 12263 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:14.457278 12263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:14.457440 12263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:14.459254 12263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "1bc773b73879473f9a403bd5a2bc559e"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-tzct"
server_key: "47b27a053371a9471b8ca78ecf93df43"
server_key_iv: "1e09f8080afe85fb990aa9896aaf7025"
server_key_version: "encryptionkey@0"
I20240501 19:56:14.459698 12263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "1bc773b73879473f9a403bd5a2bc559e"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-tzct"
server_key: "47b27a053371a9471b8ca78ecf93df43"
server_key_iv: "1e09f8080afe85fb990aa9896aaf7025"
server_key_version: "encryptionkey@0"
I20240501 19:56:14.463027 12263 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240501 19:56:14.465248 12278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.466312 12263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:14.466428 12263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "1bc773b73879473f9a403bd5a2bc559e"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-tzct"
server_key: "47b27a053371a9471b8ca78ecf93df43"
server_key_iv: "1e09f8080afe85fb990aa9896aaf7025"
server_key_version: "encryptionkey@0"
I20240501 19:56:14.466521 12263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:14.484105 12263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:14.487107 12263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:14.487269 12263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:14.487798 12263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:14.488588 12263 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:14.488636 12263 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.488688 12263 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:14.488744 12263 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.496963 12263 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:39729
I20240501 19:56:14.497009 12391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:39729 every 8 connection(s)
I20240501 19:56:14.497885 12263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 19:56:14.502899 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 12263
I20240501 19:56:14.503005 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 19:56:14.503237 8865 external_mini_cluster.cc:1428] Setting key 6d98502f195b836d31a68da4e5b9f569
May 01 19:56:14 dist-test-slave-tzct krb5kdc[11888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593374, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:14.509227 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:14.499489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:53207 (local address 127.8.168.126:39855)
0501 19:56:14.499620 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:14.499623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:14.500268 (+ 645us) server_negotiation.cc:407] Connection header received
0501 19:56:14.501482 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:14.501485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:14.501545 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:14.501616 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:14.502841 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:14.503406 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:14.504115 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:14.504265 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:14.506373 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:14.506389 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:14.506392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:14.506416 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:14.507526 (+ 1110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:14.507959 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:14.507962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:14.507963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:14.508007 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:14.508241 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:14.508245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:14.508247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:14.508388 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:14.508473 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:14.508949 (+ 476us) server_negotiation.cc:299] Negotiation successful
0501 19:56:14.509068 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20240501 19:56:14.509789 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:14.499722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39855 (local address 127.8.168.67:53207)
0501 19:56:14.500136 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:14.500169 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:14.501268 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:14.501781 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:14.501789 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:14.502157 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:14.502697 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:14.502710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:14.503514 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:14.503518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:14.503991 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:14.504002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:14.504201 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:14.504814 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:14.504848 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:14.506236 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:14.507629 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:14.507635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:14.507645 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:14.507870 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:14.508083 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:14.508086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:14.508088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:14.508165 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:14.508482 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:14.508487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:14.508700 (+ 213us) client_negotiation.cc:769] Sending connection context
0501 19:56:14.508905 (+ 205us) client_negotiation.cc:240] Negotiation successful
0501 19:56:14.509151 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
I20240501 19:56:14.510694 12392 heartbeater.cc:347] Connected to a master server at 127.8.168.126:39855
I20240501 19:56:14.510902 12392 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:14.511370 12392 heartbeater.cc:511] Master 127.8.168.126:39855 requested a full tablet report, sending...
I20240501 19:56:14.512271 11933 ts_manager.cc:194] Registered new tserver with Master: 1bc773b73879473f9a403bd5a2bc559e (127.8.168.67:39729)
I20240501 19:56:14.512856 11933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:53207
I20240501 19:56:14.516784 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 01 19:56:14 dist-test-slave-tzct krb5kdc[11888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593374, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
May 01 19:56:14 dist-test-slave-tzct krb5kdc[11888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593374, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:14.538476 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:14.531307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.8.168.126:39855)
0501 19:56:14.531473 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:14.531477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:14.531568 (+ 91us) server_negotiation.cc:407] Connection header received
0501 19:56:14.531701 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:14.531704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:14.531750 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:14.531817 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:14.532597 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:14.533134 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:14.533756 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:14.533932 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:14.535834 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:14.535851 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:14.535853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:14.535879 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:14.537065 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:14.537440 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:14.537443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:14.537444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:14.537488 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:14.537685 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:14.537688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:14.537690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:14.537868 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:14.537990 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:14.538202 (+ 212us) server_negotiation.cc:299] Negotiation successful
0501 19:56:14.538325 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20240501 19:56:14.538847 11933 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39354
I20240501 19:56:14.539615 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 11995
I20240501 19:56:14.544644 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 12129
I20240501 19:56:14.548970 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 12263
I20240501 19:56:14.553036 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 11901
2024-05-01T19:56:14Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1913 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 19:56:14 dist-test-slave-tzct krb5kdc[12404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 19:56:14 dist-test-slave-tzct krb5kdc[12404](info): set up 2 sockets
May 01 19:56:14 dist-test-slave-tzct krb5kdc[12404](info): commencing operation
krb5kdc: starting...
W20240501 19:56:15.448963 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.876s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 19:56:15 dist-test-slave-tzct krb5kdc[12404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T19:56:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 19:56:15.561766 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:42421
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:44103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:42421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:15.664631 12417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:15.664902 12417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:15.664964 12417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:15.667917 12417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:15.667966 12417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:15.667989 12417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:15.668021 12417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:15.668051 12417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:15.672048 12417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:44103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:42421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:42421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:15.672951 12417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:15.673738 12417 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:15.680402 12425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:15.680948 12422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:15.681365 12423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:15.681977 12417 server_base.cc:1034] running on GCE node
I20240501 19:56:15.682451 12417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:15.683300 12417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:15.684453 12417 hybrid_clock.cc:648] HybridClock initialized: now 1714593375684443 us; error 24 us; skew 500 ppm
May 01 19:56:15 dist-test-slave-tzct krb5kdc[12404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593375, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:15.686975 12417 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 19:56:15.687865 12417 webserver.cc:460] Webserver started at http://127.8.168.126:42903/ using document root <none> and password file <none>
I20240501 19:56:15.688359 12417 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:15.688411 12417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:15.688572 12417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:15.690297 12417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "e239c84506da486287814c8a424cb8de"
format_stamp: "Formatted at 2024-05-01 19:56:15 on dist-test-slave-tzct"
server_key: "581eb1f582b8d3768cbf2cbac14d5745"
server_key_iv: "31f4706c0ff69dbd7ceca81d966469ee"
server_key_version: "encryptionkey@0"
I20240501 19:56:15.690757 12417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "e239c84506da486287814c8a424cb8de"
format_stamp: "Formatted at 2024-05-01 19:56:15 on dist-test-slave-tzct"
server_key: "581eb1f582b8d3768cbf2cbac14d5745"
server_key_iv: "31f4706c0ff69dbd7ceca81d966469ee"
server_key_version: "encryptionkey@0"
I20240501 19:56:15.693979 12417 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:15.696152 12432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:15.697150 12417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:15.697254 12417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "e239c84506da486287814c8a424cb8de"
format_stamp: "Formatted at 2024-05-01 19:56:15 on dist-test-slave-tzct"
server_key: "581eb1f582b8d3768cbf2cbac14d5745"
server_key_iv: "31f4706c0ff69dbd7ceca81d966469ee"
server_key_version: "encryptionkey@0"
I20240501 19:56:15.697337 12417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:15.715306 12417 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:15.718358 12417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:15.718519 12417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:15.725095 12417 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:42421
I20240501 19:56:15.725102 12484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:42421 every 8 connection(s)
I20240501 19:56:15.726042 12417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:56:15.726660 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 12417
I20240501 19:56:15.726780 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:56:15.727001 8865 external_mini_cluster.cc:1428] Setting key 72349bdfa892f95ca6950690eb677d6f
I20240501 19:56:15.728940 12485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
May 01 19:56:15 dist-test-slave-tzct krb5kdc[12404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:15.734540 12485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de: Bootstrap starting.
I20240501 19:56:15.736444 12485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:15.737170 12485 log.cc:824] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:15.738709 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:15.728132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.8.168.126:42421)
0501 19:56:15.728509 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:15.728518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:15.728551 (+ 33us) server_negotiation.cc:407] Connection header received
0501 19:56:15.729162 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:15.729180 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:15.729449 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:15.729731 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:15.730688 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:15.731404 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:15.732203 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:15.732399 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:15.734781 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:15.734839 (+ 58us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:15.734849 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:15.734881 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:15.736470 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:15.736865 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:15.736869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:15.736873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:15.736943 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:15.737150 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:15.737154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:15.737155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:15.737498 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:15.737665 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:15.737944 (+ 279us) server_negotiation.cc:299] Negotiation successful
0501 19:56:15.738168 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20240501 19:56:15.738816 12485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de: No bootstrap required, opened a new log
I20240501 19:56:15.741631 12485 raft_consensus.cc:348] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e239c84506da486287814c8a424cb8de" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42421 } }
I20240501 19:56:15.741804 12485 raft_consensus.cc:374] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:15.741864 12485 raft_consensus.cc:729] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e239c84506da486287814c8a424cb8de, State: Initialized, Role: FOLLOWER
I20240501 19:56:15.742228 12485 consensus_queue.cc:260] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e239c84506da486287814c8a424cb8de" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42421 } }
I20240501 19:56:15.742344 12485 raft_consensus.cc:388] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:15.742398 12485 raft_consensus.cc:482] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:15.742496 12485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:15.743248 12485 raft_consensus.cc:504] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e239c84506da486287814c8a424cb8de" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42421 } }
I20240501 19:56:15.743536 12485 leader_election.cc:302] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e239c84506da486287814c8a424cb8de; no voters:
I20240501 19:56:15.743781 12485 leader_election.cc:288] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:15.743896 12490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:15.744076 12490 raft_consensus.cc:686] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [term 1 LEADER]: Becoming Leader. State: Replica: e239c84506da486287814c8a424cb8de, State: Running, Role: LEADER
I20240501 19:56:15.744346 12490 consensus_queue.cc:237] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e239c84506da486287814c8a424cb8de" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42421 } }
I20240501 19:56:15.744804 12485 sys_catalog.cc:564] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:15.745843 12491 sys_catalog.cc:455] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e239c84506da486287814c8a424cb8de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e239c84506da486287814c8a424cb8de" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42421 } } }
I20240501 19:56:15.745859 12492 sys_catalog.cc:455] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [sys.catalog]: SysCatalogTable state changed. Reason: New leader e239c84506da486287814c8a424cb8de. Latest consensus state: current_term: 1 leader_uuid: "e239c84506da486287814c8a424cb8de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e239c84506da486287814c8a424cb8de" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42421 } } }
I20240501 19:56:15.746012 12492 sys_catalog.cc:458] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:15.745965 12491 sys_catalog.cc:458] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:15.746382 12499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:15.748901 12499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:15.753557 12499 catalog_manager.cc:1347] Generated new cluster ID: b684917101224481bdb6b437928e008e
I20240501 19:56:15.753626 12499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:15.768117 12499 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:15.769057 12499 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:15.778159 12499 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e239c84506da486287814c8a424cb8de: Generated new TSK 0
I20240501 19:56:15.778880 12499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
I20240501 19:56:15.811636 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:42421
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:44103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:15.918725 12511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:15.918977 12511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:15.919055 12511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:15.922158 12511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:15.922235 12511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:15.922333 12511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:56:15.926512 12511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:44103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:42421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:15.927474 12511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:15.928328 12511 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:15.936136 12516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:15.936712 12517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:15.937078 12519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:15.937709 12511 server_base.cc:1034] running on GCE node
I20240501 19:56:15.938057 12511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:15.938593 12511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:15.939736 12511 hybrid_clock.cc:648] HybridClock initialized: now 1714593375939713 us; error 36 us; skew 500 ppm
May 01 19:56:15 dist-test-slave-tzct krb5kdc[12404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593375, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:15.942282 12511 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 19:56:15.943198 12511 webserver.cc:460] Webserver started at http://127.8.168.65:45767/ using document root <none> and password file <none>
I20240501 19:56:15.943714 12511 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:15.943773 12511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:15.943943 12511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:15.945897 12511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "8d4514d3f0604c47883ae83072662045"
format_stamp: "Formatted at 2024-05-01 19:56:15 on dist-test-slave-tzct"
server_key: "9b0984105cc961d5cd2d09ecdb03a4ba"
server_key_iv: "2672f62107e1de1bd7b9e12d9da00db7"
server_key_version: "encryptionkey@0"
I20240501 19:56:15.946420 12511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "8d4514d3f0604c47883ae83072662045"
format_stamp: "Formatted at 2024-05-01 19:56:15 on dist-test-slave-tzct"
server_key: "9b0984105cc961d5cd2d09ecdb03a4ba"
server_key_iv: "2672f62107e1de1bd7b9e12d9da00db7"
server_key_version: "encryptionkey@0"
I20240501 19:56:15.949677 12511 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 19:56:15.951720 12526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:15.952661 12511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:15.952790 12511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "8d4514d3f0604c47883ae83072662045"
format_stamp: "Formatted at 2024-05-01 19:56:15 on dist-test-slave-tzct"
server_key: "9b0984105cc961d5cd2d09ecdb03a4ba"
server_key_iv: "2672f62107e1de1bd7b9e12d9da00db7"
server_key_version: "encryptionkey@0"
I20240501 19:56:15.952881 12511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:15.974934 12511 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:15.977613 12511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:15.977780 12511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:15.978320 12511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:15.979113 12511 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:15.979161 12511 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:15.979213 12511 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:15.979241 12511 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:15.987408 12511 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:36179
I20240501 19:56:15.987454 12639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:36179 every 8 connection(s)
I20240501 19:56:15.988317 12511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
May 01 19:56:15 dist-test-slave-tzct krb5kdc[12404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593375, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:15.997516 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 12511
I20240501 19:56:15.997637 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 19:56:15.997869 8865 external_mini_cluster.cc:1428] Setting key b123ae3a76e34bffe70723c6f1298e90
I20240501 19:56:16.000504 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:15.990024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:45009 (local address 127.8.168.126:42421)
0501 19:56:15.990157 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:15.990161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:15.990847 (+ 686us) server_negotiation.cc:407] Connection header received
0501 19:56:15.991620 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:15.991623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:15.991673 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:15.991750 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:15.993436 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:15.993947 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:15.994531 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:15.994657 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:15.996805 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:15.996827 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:15.996829 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:15.996853 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:15.998258 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:15.998804 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:15.998807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:15.998809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:15.998855 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:15.999219 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:15.999221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:15.999223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:15.999396 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:15.999497 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:16.000148 (+ 651us) server_negotiation.cc:299] Negotiation successful
0501 19:56:16.000305 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20240501 19:56:16.001215 12642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:15.990298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:42421 (local address 127.8.168.65:45009)
0501 19:56:15.990698 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:15.990734 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:15.991414 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:15.991898 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:15.991905 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:15.992285 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:15.993281 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:15.993306 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:15.994048 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:15.994051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:15.994449 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:15.994456 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:15.994627 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:15.995212 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:15.995240 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:15.996644 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:15.998398 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:15.998404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:15.998415 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:15.998711 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:15.998933 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:15.998936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:15.998938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:15.999032 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:15.999513 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:15.999520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:15.999862 (+ 342us) client_negotiation.cc:769] Sending connection context
0501 19:56:16.000074 (+ 212us) client_negotiation.cc:240] Negotiation successful
0501 19:56:16.000352 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":89,"threads_started":1}
I20240501 19:56:16.002384 12640 heartbeater.cc:347] Connected to a master server at 127.8.168.126:42421
I20240501 19:56:16.002605 12640 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:16.003122 12640 heartbeater.cc:511] Master 127.8.168.126:42421 requested a full tablet report, sending...
I20240501 19:56:16.004528 12449 ts_manager.cc:194] Registered new tserver with Master: 8d4514d3f0604c47883ae83072662045 (127.8.168.65:36179)
I20240501 19:56:16.005681 12449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:45009
WARNING: no policy specified for kudu/127.8.168.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
I20240501 19:56:16.022899 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:42421
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:44103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:16.128902 12645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:16.129130 12645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:16.129191 12645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:16.132244 12645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:16.132310 12645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:16.132390 12645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:56:16.136637 12645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:44103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:42421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:16.137521 12645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:16.138335 12645 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:16.146281 12650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:16.146796 12651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:16.147352 12653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:16.147943 12645 server_base.cc:1034] running on GCE node
I20240501 19:56:16.148320 12645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:16.148977 12645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:16.150123 12645 hybrid_clock.cc:648] HybridClock initialized: now 1714593376150110 us; error 27 us; skew 500 ppm
May 01 19:56:16 dist-test-slave-tzct krb5kdc[12404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593376, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:16.152827 12645 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 19:56:16.153743 12645 webserver.cc:460] Webserver started at http://127.8.168.66:35549/ using document root <none> and password file <none>
I20240501 19:56:16.154278 12645 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:16.154343 12645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:16.154531 12645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:16.156420 12645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "1de48a97caf4483fb5f8130709fb1ecc"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-tzct"
server_key: "3f071fc1fd88058963cdc128d3928695"
server_key_iv: "2024cfc2381d5be332da5e477bf3c0c5"
server_key_version: "encryptionkey@0"
I20240501 19:56:16.156934 12645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "1de48a97caf4483fb5f8130709fb1ecc"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-tzct"
server_key: "3f071fc1fd88058963cdc128d3928695"
server_key_iv: "2024cfc2381d5be332da5e477bf3c0c5"
server_key_version: "encryptionkey@0"
I20240501 19:56:16.160319 12645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 19:56:16.162694 12660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.163724 12645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:16.163878 12645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "1de48a97caf4483fb5f8130709fb1ecc"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-tzct"
server_key: "3f071fc1fd88058963cdc128d3928695"
server_key_iv: "2024cfc2381d5be332da5e477bf3c0c5"
server_key_version: "encryptionkey@0"
I20240501 19:56:16.163973 12645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:16.180488 12645 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:16.183730 12645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:16.183913 12645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:16.184466 12645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:16.185344 12645 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:16.185391 12645 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.185442 12645 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:16.185469 12645 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.194006 12645 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:46487
I20240501 19:56:16.194051 12773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:46487 every 8 connection(s)
I20240501 19:56:16.194995 12645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 19:56:16.198594 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 12645
I20240501 19:56:16.198712 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 19:56:16.198997 8865 external_mini_cluster.cc:1428] Setting key 152d35ebd7a22fa349e7eb02f9b8acbf
May 01 19:56:16 dist-test-slave-tzct krb5kdc[12404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593376, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.168.67@KRBTEST.COM; defaulting to no policy
I20240501 19:56:16.208302 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:16.196938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:36749 (local address 127.8.168.126:42421)
0501 19:56:16.197097 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:16.197102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:16.197807 (+ 705us) server_negotiation.cc:407] Connection header received
0501 19:56:16.198616 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:16.198620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:16.198679 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:16.198750 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:16.200594 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:16.201336 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:16.202085 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:16.202234 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:16.204791 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:16.204814 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:16.204816 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:16.204842 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:16.206221 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:16.206779 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:16.206782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:16.206783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:16.206822 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:16.207153 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:16.207156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:16.207158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:16.207341 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:16.207440 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:16.208039 (+ 599us) server_negotiation.cc:299] Negotiation successful
0501 19:56:16.208169 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20240501 19:56:16.208943 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:16.197221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:42421 (local address 127.8.168.66:36749)
0501 19:56:16.197664 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:16.197703 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:16.198408 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:16.198986 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:16.198994 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:16.199401 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:16.200465 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:16.200480 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:16.201449 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:16.201452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:16.201959 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:16.201970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:16.202179 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:16.202794 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:16.202829 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:16.204492 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:16.206348 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:16.206354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:16.206366 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:16.206686 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:16.206899 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:16.206903 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:16.206905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:16.207030 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:16.207456 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:16.207463 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:16.207789 (+ 326us) client_negotiation.cc:769] Sending connection context
0501 19:56:16.207967 (+ 178us) client_negotiation.cc:240] Negotiation successful
0501 19:56:16.208197 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.67@KRBTEST.COM" created.
I20240501 19:56:16.210331 12774 heartbeater.cc:347] Connected to a master server at 127.8.168.126:42421
I20240501 19:56:16.210570 12774 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:16.211114 12774 heartbeater.cc:511] Master 127.8.168.126:42421 requested a full tablet report, sending...
I20240501 19:56:16.212293 12449 ts_manager.cc:194] Registered new tserver with Master: 1de48a97caf4483fb5f8130709fb1ecc (127.8.168.66:46487)
I20240501 19:56:16.213014 12449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:36749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
I20240501 19:56:16.225889 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:42421
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:44103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:16.334726 12779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:16.334964 12779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:16.335045 12779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:16.338146 12779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:16.338212 12779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:16.338311 12779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:56:16.342839 12779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:44103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:42421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:16.343808 12779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:16.344642 12779 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:16.352949 12784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:16.353396 12787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:16.353986 12785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:16.354516 12779 server_base.cc:1034] running on GCE node
I20240501 19:56:16.354914 12779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:16.355486 12779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:16.356623 12779 hybrid_clock.cc:648] HybridClock initialized: now 1714593376356610 us; error 29 us; skew 500 ppm
May 01 19:56:16 dist-test-slave-tzct krb5kdc[12404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593376, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:16.359313 12779 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 19:56:16.360272 12779 webserver.cc:460] Webserver started at http://127.8.168.67:34045/ using document root <none> and password file <none>
I20240501 19:56:16.360853 12779 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:16.360920 12779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:16.361109 12779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:16.362915 12779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "3b75f6c3d18849498de2694401fad0d6"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-tzct"
server_key: "b8d8d5c34fabfd13f4158bfeed5b433d"
server_key_iv: "a30ab27f0a849452706e6a959de26838"
server_key_version: "encryptionkey@0"
I20240501 19:56:16.363380 12779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "3b75f6c3d18849498de2694401fad0d6"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-tzct"
server_key: "b8d8d5c34fabfd13f4158bfeed5b433d"
server_key_iv: "a30ab27f0a849452706e6a959de26838"
server_key_version: "encryptionkey@0"
I20240501 19:56:16.366731 12779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:16.369102 12794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.370151 12779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:16.370261 12779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "3b75f6c3d18849498de2694401fad0d6"
format_stamp: "Formatted at 2024-05-01 19:56:16 on dist-test-slave-tzct"
server_key: "b8d8d5c34fabfd13f4158bfeed5b433d"
server_key_iv: "a30ab27f0a849452706e6a959de26838"
server_key_version: "encryptionkey@0"
I20240501 19:56:16.370348 12779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:16.391330 12779 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:16.394294 12779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:16.394464 12779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:16.395007 12779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:16.395799 12779 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:16.395846 12779 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.395905 12779 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:16.395928 12779 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:16.404506 12779 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:35619
I20240501 19:56:16.404577 12907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:35619 every 8 connection(s)
I20240501 19:56:16.405483 12779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 19:56:16.411638 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 12779
I20240501 19:56:16.411756 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 19:56:16.411980 8865 external_mini_cluster.cc:1428] Setting key 92f2ffe96581d739de3fa1d4c7716917
May 01 19:56:16 dist-test-slave-tzct krb5kdc[12404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593376, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:16.417213 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:16.407141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:37041 (local address 127.8.168.126:42421)
0501 19:56:16.407280 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:16.407283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:16.407935 (+ 652us) server_negotiation.cc:407] Connection header received
0501 19:56:16.408749 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:16.408753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:16.408814 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:16.408890 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:16.410503 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:16.411003 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:16.411628 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:16.411806 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:16.414298 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:16.414317 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:16.414321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:16.414367 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:16.415556 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:16.415997 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:16.415999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:16.416000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:16.416036 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:16.416231 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:16.416233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:16.416234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:16.416386 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:16.416452 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:16.416981 (+ 529us) server_negotiation.cc:299] Negotiation successful
0501 19:56:16.417083 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20240501 19:56:16.417889 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:16.407378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:42421 (local address 127.8.168.67:37041)
0501 19:56:16.407772 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:16.407808 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:16.408521 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:16.409065 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:16.409073 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:16.409449 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:16.410354 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:16.410368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:16.411118 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:16.411121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:16.411511 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:16.411519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:16.411737 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:16.412375 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:16.412403 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:16.414121 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:16.415661 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:16.415665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:16.415674 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:16.415919 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:16.416097 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:16.416099 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:16.416101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:16.416174 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:16.416468 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:16.416472 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:16.416714 (+ 242us) client_negotiation.cc:769] Sending connection context
0501 19:56:16.416953 (+ 239us) client_negotiation.cc:240] Negotiation successful
0501 19:56:16.417209 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":88,"threads_started":1}
I20240501 19:56:16.418901 12908 heartbeater.cc:347] Connected to a master server at 127.8.168.126:42421
I20240501 19:56:16.419126 12908 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:16.419623 12908 heartbeater.cc:511] Master 127.8.168.126:42421 requested a full tablet report, sending...
I20240501 19:56:16.420766 12449 ts_manager.cc:194] Registered new tserver with Master: 3b75f6c3d18849498de2694401fad0d6 (127.8.168.67:35619)
I20240501 19:56:16.421342 12449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:37041
I20240501 19:56:16.426018 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 01 19:56:16 dist-test-slave-tzct krb5kdc[12404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 01 19:56:16 dist-test-slave-tzct krb5kdc[12404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.65@KRBTEST.COM
I20240501 19:56:16.449002 12915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:16.440653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37672 (local address 127.8.168.65:36179)
0501 19:56:16.441021 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:16.441026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:16.441050 (+ 24us) server_negotiation.cc:407] Connection header received
0501 19:56:16.441106 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:16.441111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:16.441268 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:16.441387 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:16.442121 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:16.442827 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:16.443465 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:16.443636 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:16.445649 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:16.445679 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:16.445691 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:16.445717 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:16.447481 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:16.447895 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:16.447901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:16.447902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:16.447967 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:16.448145 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:16.448146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:16.448147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:16.448345 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:16.448499 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:16.448666 (+ 167us) server_negotiation.cc:299] Negotiation successful
0501 19:56:16.448857 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":136,"threads_started":1}
May 01 19:56:16 dist-test-slave-tzct krb5kdc[12404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:16.458405 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:16.451595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.8.168.126:42421)
0501 19:56:16.451728 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:16.451732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:16.451745 (+ 13us) server_negotiation.cc:407] Connection header received
0501 19:56:16.451811 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:16.451814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:16.451858 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:16.451923 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:16.452544 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:16.453087 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:16.453632 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:16.453773 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:16.455662 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:16.455682 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:16.455684 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:16.455709 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:16.457183 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:16.457517 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:16.457520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:16.457521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:16.457561 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:16.457710 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:16.457712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:16.457713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:16.457898 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:16.457998 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:16.458148 (+ 150us) server_negotiation.cc:299] Negotiation successful
0501 19:56:16.458241 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20240501 19:56:16.458837 12449 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32850
I20240501 19:56:16.459443 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 12511
I20240501 19:56:16.464839 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 12645
I20240501 19:56:16.469082 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 12779
I20240501 19:56:16.473693 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 12417
2024-05-01T19:56:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1920 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 19:56:16 dist-test-slave-tzct krb5kdc[12920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 19:56:16 dist-test-slave-tzct krb5kdc[12920](info): set up 2 sockets
May 01 19:56:16 dist-test-slave-tzct krb5kdc[12920](info): commencing operation
krb5kdc: starting...
W20240501 19:56:17.360991 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.867s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 19:56:17 dist-test-slave-tzct krb5kdc[12920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T19:56:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 19:56:17.471243 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:37525
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:37525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:17.575024 12933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:17.575276 12933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:17.575332 12933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:17.578667 12933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:17.578742 12933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:17.578764 12933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:17.578792 12933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:17.578816 12933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:17.583061 12933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:37525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:37525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:17.583891 12933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:17.584669 12933 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:17.591501 12938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:17.592073 12939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:17.592613 12941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:17.593114 12933 server_base.cc:1034] running on GCE node
I20240501 19:56:17.593619 12933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:17.594503 12933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:17.595675 12933 hybrid_clock.cc:648] HybridClock initialized: now 1714593377595650 us; error 37 us; skew 500 ppm
May 01 19:56:17 dist-test-slave-tzct krb5kdc[12920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593377, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:17.598214 12933 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 19:56:17.598914 12933 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:17.598978 12933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:17.599144 12933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:17.600930 12933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "12d754dfa45c48c8869a00d6ce7864ce"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-tzct"
server_key: "f1d10f6d105d1bc6f7c0c337e8f48ce3"
server_key_iv: "96c2e8cb889f9300f63b86dfc37d602a"
server_key_version: "encryptionkey@0"
I20240501 19:56:17.601397 12933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "12d754dfa45c48c8869a00d6ce7864ce"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-tzct"
server_key: "f1d10f6d105d1bc6f7c0c337e8f48ce3"
server_key_iv: "96c2e8cb889f9300f63b86dfc37d602a"
server_key_version: "encryptionkey@0"
I20240501 19:56:17.604563 12933 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:17.606681 12947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:17.607754 12933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:17.607882 12933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "12d754dfa45c48c8869a00d6ce7864ce"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-tzct"
server_key: "f1d10f6d105d1bc6f7c0c337e8f48ce3"
server_key_iv: "96c2e8cb889f9300f63b86dfc37d602a"
server_key_version: "encryptionkey@0"
I20240501 19:56:17.607968 12933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:17.629781 12933 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:17.632886 12933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:17.633050 12933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:17.639451 12999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:37525 every 8 connection(s)
I20240501 19:56:17.639458 12933 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:37525
I20240501 19:56:17.640062 12933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:56:17.642796 13000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:17.646329 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 12933
I20240501 19:56:17.646446 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:56:17.646696 8865 external_mini_cluster.cc:1428] Setting key dbfb25473a7731ecddeae91dc2dea6c9
I20240501 19:56:17.647890 13000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce: Bootstrap starting.
I20240501 19:56:17.650082 13000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:17.650889 13000 log.cc:824] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:17.652812 13000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce: No bootstrap required, opened a new log
May 01 19:56:17 dist-test-slave-tzct krb5kdc[12920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:17.656245 13000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d754dfa45c48c8869a00d6ce7864ce" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 37525 } }
I20240501 19:56:17.656468 13000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:17.656533 13000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d754dfa45c48c8869a00d6ce7864ce, State: Initialized, Role: FOLLOWER
I20240501 19:56:17.657025 13000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d754dfa45c48c8869a00d6ce7864ce" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 37525 } }
I20240501 19:56:17.657176 13000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:17.657243 13000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:17.657341 13000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:17.658277 13000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d754dfa45c48c8869a00d6ce7864ce" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 37525 } }
I20240501 19:56:17.658633 13000 leader_election.cc:302] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12d754dfa45c48c8869a00d6ce7864ce; no voters:
I20240501 19:56:17.658947 13000 leader_election.cc:288] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:17.659052 13005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:17.659305 13005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [term 1 LEADER]: Becoming Leader. State: Replica: 12d754dfa45c48c8869a00d6ce7864ce, State: Running, Role: LEADER
I20240501 19:56:17.659638 13005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d754dfa45c48c8869a00d6ce7864ce" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 37525 } }
I20240501 19:56:17.659857 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:17.647944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59898 (local address 127.8.168.126:37525)
0501 19:56:17.648363 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:17.648372 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:17.648409 (+ 37us) server_negotiation.cc:407] Connection header received
0501 19:56:17.649103 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:17.649122 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:17.649416 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:17.649721 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:17.650806 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:17.651600 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:17.652426 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:17.652643 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:17.655017 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:17.655042 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:17.655059 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:17.655107 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:17.657092 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:17.657543 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:17.657548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:17.657553 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:17.657646 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:17.657967 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:17.657970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:17.657972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:17.658430 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:17.658631 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:17.658863 (+ 232us) server_negotiation.cc:299] Negotiation successful
0501 19:56:17.659129 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":134,"threads_started":1}
I20240501 19:56:17.660137 13000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:17.661156 13006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12d754dfa45c48c8869a00d6ce7864ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d754dfa45c48c8869a00d6ce7864ce" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 37525 } } }
I20240501 19:56:17.661267 13006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:17.661679 13007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12d754dfa45c48c8869a00d6ce7864ce. Latest consensus state: current_term: 1 leader_uuid: "12d754dfa45c48c8869a00d6ce7864ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12d754dfa45c48c8869a00d6ce7864ce" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 37525 } } }
I20240501 19:56:17.661851 13007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce [sys.catalog]: This master's current role is: LEADER
W20240501 19:56:17.664880 13021 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:56:17.664955 13021 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:56:17.665009 13014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:17.665846 13014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:17.670758 13014 catalog_manager.cc:1347] Generated new cluster ID: 7622962ca5734c10908cb0caa5e7b105
I20240501 19:56:17.670827 13014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:17.697710 13014 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:17.698673 13014 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:17.705178 13014 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce: Generated new TSK 0
I20240501 19:56:17.705864 13014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
I20240501 19:56:17.737936 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:37525
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:17.843319 13026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:17.843549 13026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:17.843607 13026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:17.846619 13026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:17.846684 13026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:17.846769 13026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:56:17.850841 13026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:37525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:17.851675 13026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:17.852553 13026 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:17.860031 13031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:17.860543 13032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:17.861037 13034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:17.861552 13026 server_base.cc:1034] running on GCE node
I20240501 19:56:17.862016 13026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:17.862609 13026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:17.863807 13026 hybrid_clock.cc:648] HybridClock initialized: now 1714593377863785 us; error 34 us; skew 500 ppm
May 01 19:56:17 dist-test-slave-tzct krb5kdc[12920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593377, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:17.866371 13026 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 19:56:17.867094 13026 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:17.867156 13026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:17.867342 13026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:17.869232 13026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "8f4169a210e048d9815dc6923f1d39e6"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-tzct"
server_key: "2b77af68d12f6e1258c2d9e8139800e8"
server_key_iv: "6fd6b72bc1b028d9398870249c081c86"
server_key_version: "encryptionkey@0"
I20240501 19:56:17.869711 13026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "8f4169a210e048d9815dc6923f1d39e6"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-tzct"
server_key: "2b77af68d12f6e1258c2d9e8139800e8"
server_key_iv: "6fd6b72bc1b028d9398870249c081c86"
server_key_version: "encryptionkey@0"
I20240501 19:56:17.873167 13026 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:17.875336 13040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:17.876300 13026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:17.876401 13026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "8f4169a210e048d9815dc6923f1d39e6"
format_stamp: "Formatted at 2024-05-01 19:56:17 on dist-test-slave-tzct"
server_key: "2b77af68d12f6e1258c2d9e8139800e8"
server_key_iv: "6fd6b72bc1b028d9398870249c081c86"
server_key_version: "encryptionkey@0"
I20240501 19:56:17.876497 13026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:17.894677 13026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:17.897437 13026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:17.897604 13026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:17.898141 13026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:17.898954 13026 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:17.899000 13026 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:17.899042 13026 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:17.899086 13026 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:17.907478 13026 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:39727
I20240501 19:56:17.907496 13153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:39727 every 8 connection(s)
I20240501 19:56:17.908162 13026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 19:56:17.913087 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 13026
I20240501 19:56:17.913208 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 19:56:17.913439 8865 external_mini_cluster.cc:1428] Setting key 015d8542fb05443872e8f3c239b22ac2
May 01 19:56:17 dist-test-slave-tzct krb5kdc[12920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593377, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:17.920295 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:17.909816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:56505 (local address 127.8.168.126:37525)
0501 19:56:17.909947 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:17.909951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:17.910588 (+ 637us) server_negotiation.cc:407] Connection header received
0501 19:56:17.911412 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:17.911415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:17.911470 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:17.911545 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:17.913194 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:17.913739 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:17.914408 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:17.914588 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:17.916879 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:17.916899 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:17.916902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:17.916928 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:17.918360 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:17.918825 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:17.918828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:17.918837 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:17.918896 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:17.919148 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:17.919151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:17.919154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:17.919349 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:17.919450 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:17.919989 (+ 539us) server_negotiation.cc:299] Negotiation successful
0501 19:56:17.920124 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240501 19:56:17.920898 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:17.910053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:37525 (local address 127.8.168.65:56505)
0501 19:56:17.910461 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:17.910496 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:17.911210 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:17.911707 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:17.911716 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:17.912102 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:17.913048 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:17.913062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:17.913852 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:17.913855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:17.914297 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:17.914306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:17.914512 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:17.915107 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:17.915142 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:17.916686 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:17.918470 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:17.918474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:17.918483 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:17.918719 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:17.918984 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:17.918986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:17.918987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:17.919074 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:17.919472 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:17.919477 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:17.919714 (+ 237us) client_negotiation.cc:769] Sending connection context
0501 19:56:17.919922 (+ 208us) client_negotiation.cc:240] Negotiation successful
0501 19:56:17.920151 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.8.168.66@KRBTEST.COM; defaulting to no policy
I20240501 19:56:17.922041 13154 heartbeater.cc:347] Connected to a master server at 127.8.168.126:37525
I20240501 19:56:17.922252 13154 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:17.922719 13154 heartbeater.cc:511] Master 127.8.168.126:37525 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.66@KRBTEST.COM" created.
I20240501 19:56:17.924203 12963 ts_manager.cc:194] Registered new tserver with Master: 8f4169a210e048d9815dc6923f1d39e6 (127.8.168.65:39727)
I20240501 19:56:17.925566 12963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:56505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
I20240501 19:56:17.937629 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:37525
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:18.040016 13159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:18.040238 13159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:18.040295 13159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:18.043299 13159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:18.043359 13159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:18.043442 13159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:56:18.047447 13159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:37525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:18.048281 13159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:18.049090 13159 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:18.056578 13165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:18.057004 13164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:18.057614 13167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:18.058039 13159 server_base.cc:1034] running on GCE node
I20240501 19:56:18.058398 13159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:18.058924 13159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:18.060072 13159 hybrid_clock.cc:648] HybridClock initialized: now 1714593378060059 us; error 25 us; skew 500 ppm
May 01 19:56:18 dist-test-slave-tzct krb5kdc[12920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593378, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:18.062549 13159 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 19:56:18.063223 13159 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:18.063277 13159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:18.063436 13159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:18.065173 13159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "213f39f01d8346f2be49410119a53900"
format_stamp: "Formatted at 2024-05-01 19:56:18 on dist-test-slave-tzct"
server_key: "8fc52bd3fbe856e9e367bb3e353b0c58"
server_key_iv: "616d47abdb46b85bc7b7eb83d429f047"
server_key_version: "encryptionkey@0"
I20240501 19:56:18.065598 13159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "213f39f01d8346f2be49410119a53900"
format_stamp: "Formatted at 2024-05-01 19:56:18 on dist-test-slave-tzct"
server_key: "8fc52bd3fbe856e9e367bb3e353b0c58"
server_key_iv: "616d47abdb46b85bc7b7eb83d429f047"
server_key_version: "encryptionkey@0"
I20240501 19:56:18.068607 13159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:18.070596 13173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:18.071516 13159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:18.071607 13159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "213f39f01d8346f2be49410119a53900"
format_stamp: "Formatted at 2024-05-01 19:56:18 on dist-test-slave-tzct"
server_key: "8fc52bd3fbe856e9e367bb3e353b0c58"
server_key_iv: "616d47abdb46b85bc7b7eb83d429f047"
server_key_version: "encryptionkey@0"
I20240501 19:56:18.071693 13159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:18.084642 13159 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:18.087117 13159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:18.087265 13159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:18.087762 13159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:18.088579 13159 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:18.088625 13159 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:18.088667 13159 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:18.088696 13159 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:18.096843 13159 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:36717
I20240501 19:56:18.096932 13286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:36717 every 8 connection(s)
I20240501 19:56:18.097358 13159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 19:56:18.102542 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 13159
I20240501 19:56:18.102641 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 19:56:18.102877 8865 external_mini_cluster.cc:1428] Setting key a5ef01f9d1c27cc3c94d91141f112672
May 01 19:56:18 dist-test-slave-tzct krb5kdc[12920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593378, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:18.109217 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.098862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:46699 (local address 127.8.168.126:37525)
0501 19:56:18.099067 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:18.099070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:18.099603 (+ 533us) server_negotiation.cc:407] Connection header received
0501 19:56:18.100441 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:18.100446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:18.100508 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:18.100592 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:18.102212 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.102796 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:18.103546 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.103728 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:18.106073 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:18.106091 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:18.106093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:18.106115 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:18.107496 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:18.107924 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:18.107927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:18.107929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:18.107980 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:18.108218 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:18.108221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:18.108223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:18.108369 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:18.108454 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:18.108938 (+ 484us) server_negotiation.cc:299] Negotiation successful
0501 19:56:18.109061 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20240501 19:56:18.109778 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.099064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:37525 (local address 127.8.168.66:46699)
0501 19:56:18.099466 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:18.099501 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:18.100207 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:18.100779 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:18.100790 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:18.101166 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:18.102061 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:18.102076 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.102914 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:18.102919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:18.103422 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:18.103432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.103637 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:18.104357 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:18.104385 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:18.105931 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:18.107597 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:18.107602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:18.107612 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:18.107835 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:18.108057 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:18.108060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:18.108062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:18.108140 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:18.108463 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:18.108480 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:18.108692 (+ 212us) client_negotiation.cc:769] Sending connection context
0501 19:56:18.108893 (+ 201us) client_negotiation.cc:240] Negotiation successful
0501 19:56:18.109145 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.8.168.67@KRBTEST.COM; defaulting to no policy
I20240501 19:56:18.110893 13287 heartbeater.cc:347] Connected to a master server at 127.8.168.126:37525
I20240501 19:56:18.111115 13287 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:18.111544 13287 heartbeater.cc:511] Master 127.8.168.126:37525 requested a full tablet report, sending...
I20240501 19:56:18.112493 12963 ts_manager.cc:194] Registered new tserver with Master: 213f39f01d8346f2be49410119a53900 (127.8.168.66:36717)
I20240501 19:56:18.113095 12963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:46699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
I20240501 19:56:18.125505 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:37525
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:18.227902 13292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:18.228103 13292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:18.228176 13292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:18.231149 13292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:18.231210 13292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:18.231307 13292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:56:18.235294 13292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:37525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:18.236220 13292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:18.237128 13292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:18.245805 13297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:18.246163 13300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:18.246786 13298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:18.247291 13292 server_base.cc:1034] running on GCE node
I20240501 19:56:18.247606 13292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:18.248136 13292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:18.249269 13292 hybrid_clock.cc:648] HybridClock initialized: now 1714593378249258 us; error 39 us; skew 500 ppm
May 01 19:56:18 dist-test-slave-tzct krb5kdc[12920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593378, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:18.251685 13292 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 19:56:18.252341 13292 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:18.252399 13292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:18.252578 13292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:18.254405 13292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "6514607627a74f67adf25a3fd2031aff"
format_stamp: "Formatted at 2024-05-01 19:56:18 on dist-test-slave-tzct"
server_key: "3caa9b0a9c269dd5eb6b5c53d8659b65"
server_key_iv: "b5f4664cef2b679b35adc81a9a47aeb2"
server_key_version: "encryptionkey@0"
I20240501 19:56:18.254828 13292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "6514607627a74f67adf25a3fd2031aff"
format_stamp: "Formatted at 2024-05-01 19:56:18 on dist-test-slave-tzct"
server_key: "3caa9b0a9c269dd5eb6b5c53d8659b65"
server_key_iv: "b5f4664cef2b679b35adc81a9a47aeb2"
server_key_version: "encryptionkey@0"
I20240501 19:56:18.258000 13292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:18.260004 13306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:18.261040 13292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:18.261144 13292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "6514607627a74f67adf25a3fd2031aff"
format_stamp: "Formatted at 2024-05-01 19:56:18 on dist-test-slave-tzct"
server_key: "3caa9b0a9c269dd5eb6b5c53d8659b65"
server_key_iv: "b5f4664cef2b679b35adc81a9a47aeb2"
server_key_version: "encryptionkey@0"
I20240501 19:56:18.261225 13292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:18.274583 13292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:18.277567 13292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:18.277724 13292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:18.278231 13292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:18.278999 13292 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:18.279044 13292 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:18.279096 13292 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:18.279120 13292 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:18.287292 13292 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:44301
I20240501 19:56:18.287320 13419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:44301 every 8 connection(s)
I20240501 19:56:18.287781 13292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 19:56:18.290318 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 13292
I20240501 19:56:18.290421 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 19:56:18.290632 8865 external_mini_cluster.cc:1428] Setting key 1680b120b60cb7ffc1417679f24fb14f
May 01 19:56:18 dist-test-slave-tzct krb5kdc[12920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593378, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:18.298985 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.289336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:39723 (local address 127.8.168.126:37525)
0501 19:56:18.289467 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:18.289470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:18.290090 (+ 620us) server_negotiation.cc:407] Connection header received
0501 19:56:18.290919 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:18.290922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:18.290980 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:18.291053 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:18.292597 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.293138 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:18.293734 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.293910 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:18.295901 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:18.295917 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:18.295919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:18.295944 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:18.297245 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:18.297688 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:18.297692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:18.297693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:18.297738 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:18.297981 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:18.297984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:18.297986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:18.298134 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:18.298215 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:18.298691 (+ 476us) server_negotiation.cc:299] Negotiation successful
0501 19:56:18.298811 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20240501 19:56:18.299573 13422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.289585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:37525 (local address 127.8.168.67:39723)
0501 19:56:18.289965 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:18.289999 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:18.290722 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:18.291222 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:18.291229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:18.291586 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:18.292451 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:18.292464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.293245 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:18.293247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:18.293635 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:18.293642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.293840 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:18.294409 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:18.294434 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:18.295750 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:18.297362 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:18.297368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:18.297380 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:18.297601 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:18.297815 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:18.297818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:18.297820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:18.297903 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:18.298229 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:18.298235 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:18.298468 (+ 233us) client_negotiation.cc:769] Sending connection context
0501 19:56:18.298643 (+ 175us) client_negotiation.cc:240] Negotiation successful
0501 19:56:18.298910 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":94,"threads_started":1}
I20240501 19:56:18.300711 13420 heartbeater.cc:347] Connected to a master server at 127.8.168.126:37525
I20240501 19:56:18.301010 13420 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:18.301507 13420 heartbeater.cc:511] Master 127.8.168.126:37525 requested a full tablet report, sending...
I20240501 19:56:18.302542 12963 ts_manager.cc:194] Registered new tserver with Master: 6514607627a74f67adf25a3fd2031aff (127.8.168.67:44301)
I20240501 19:56:18.303194 12963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:39723
I20240501 19:56:18.304215 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 19:56:18.312841 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.306149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59902 (local address 127.8.168.126:37525)
0501 19:56:18.306308 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:18.306313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:18.306429 (+ 116us) server_negotiation.cc:407] Connection header received
0501 19:56:18.306554 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:18.306557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:18.306597 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:18.306661 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:18.307445 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.307941 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:18.308566 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.308773 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:18.309831 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:18.309848 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:18.309850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:18.309876 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:18.311303 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:18.311689 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:18.311693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:18.311694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:18.311744 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:18.311945 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:18.311949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:18.311951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:18.312181 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:18.312313 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:18.312528 (+ 215us) server_negotiation.cc:299] Negotiation successful
0501 19:56:18.312645 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20240501 19:56:18.315698 12963 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:56:18.318240 12963 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:18.330646 13433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.326114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:44301 (local address 127.0.0.1:41414)
0501 19:56:18.326771 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:18.326813 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:18.326985 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:18.327850 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:18.327856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:18.327878 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:18.328178 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:18.328188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.329384 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:18.329388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:18.330081 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:18.330092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.330196 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:18.330275 (+ 79us) client_negotiation.cc:769] Sending connection context
0501 19:56:18.330379 (+ 104us) client_negotiation.cc:240] Negotiation successful
0501 19:56:18.330474 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":88,"threads_started":1}
I20240501 19:56:18.331476 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.326250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.8.168.67:44301)
0501 19:56:18.327320 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:18.327327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:18.327353 (+ 26us) server_negotiation.cc:407] Connection header received
0501 19:56:18.327459 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:18.327465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:18.327662 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:18.327840 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:18.328330 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.329276 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:18.330710 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.331109 (+ 399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:18.331204 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:18.331269 (+ 65us) server_negotiation.cc:299] Negotiation successful
0501 19:56:18.331335 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961,"thread_start_us":108,"threads_started":1}
I20240501 19:56:18.331514 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.326545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50946 (local address 127.8.168.66:36717)
0501 19:56:18.326901 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:18.326907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:18.327527 (+ 620us) server_negotiation.cc:407] Connection header received
0501 19:56:18.327625 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:18.327631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:18.327787 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:18.327920 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:18.328315 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.329252 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:18.330435 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.330943 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:18.331065 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:18.331273 (+ 208us) server_negotiation.cc:299] Negotiation successful
0501 19:56:18.331371 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":94,"threads_started":1}
I20240501 19:56:18.331904 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.326394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:36717 (local address 127.0.0.1:50946)
0501 19:56:18.327339 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:18.327351 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:18.327435 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:18.327915 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:18.327918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:18.327934 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:18.328179 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:18.328187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.329393 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:18.329396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:18.330284 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:18.330292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.331122 (+ 830us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:18.331137 (+ 15us) client_negotiation.cc:769] Sending connection context
0501 19:56:18.331693 (+ 556us) client_negotiation.cc:240] Negotiation successful
0501 19:56:18.331753 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":96,"threads_started":1}
I20240501 19:56:18.331912 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.326800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:39727 (local address 127.0.0.1:48274)
0501 19:56:18.328109 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:18.328123 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:18.328204 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:18.328631 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:18.328635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:18.328655 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:18.328912 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:18.328922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.330202 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:18.330205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:18.331584 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:18.331595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.331699 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:18.331715 (+ 16us) client_negotiation.cc:769] Sending connection context
0501 19:56:18.331761 (+ 46us) client_negotiation.cc:240] Negotiation successful
0501 19:56:18.331802 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1240,"thread_start_us":94,"threads_started":1}
I20240501 19:56:18.333021 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.327155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.8.168.65:39727)
0501 19:56:18.328204 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:18.328211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:18.328232 (+ 21us) server_negotiation.cc:407] Connection header received
0501 19:56:18.328328 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:18.328333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:18.328492 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:18.328622 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:18.329042 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.330082 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:18.331981 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.332622 (+ 641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:18.332752 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:18.332822 (+ 70us) server_negotiation.cc:299] Negotiation successful
0501 19:56:18.332915 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":953,"thread_start_us":111,"threads_started":1}
I20240501 19:56:18.334287 13354 tablet_service.cc:1467] Processing CreateTablet for tablet 4fbc63098a1a44a09eafc520aec45c29 (DEFAULT_TABLE table=test-table [id=b5887c187fc74d6bb438f0997ca953f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:18.334288 13221 tablet_service.cc:1467] Processing CreateTablet for tablet 4fbc63098a1a44a09eafc520aec45c29 (DEFAULT_TABLE table=test-table [id=b5887c187fc74d6bb438f0997ca953f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:18.335323 13221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fbc63098a1a44a09eafc520aec45c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:18.335336 13354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fbc63098a1a44a09eafc520aec45c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:18.335214 13088 tablet_service.cc:1467] Processing CreateTablet for tablet 4fbc63098a1a44a09eafc520aec45c29 (DEFAULT_TABLE table=test-table [id=b5887c187fc74d6bb438f0997ca953f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:18.335958 13088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fbc63098a1a44a09eafc520aec45c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:18.342442 13439 tablet_bootstrap.cc:492] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff: Bootstrap starting.
I20240501 19:56:18.342432 13438 tablet_bootstrap.cc:492] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900: Bootstrap starting.
I20240501 19:56:18.342902 13437 tablet_bootstrap.cc:492] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6: Bootstrap starting.
I20240501 19:56:18.344655 13437 tablet_bootstrap.cc:654] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:18.344681 13438 tablet_bootstrap.cc:654] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:18.344681 13439 tablet_bootstrap.cc:654] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:18.345479 13437 log.cc:824] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:18.345544 13439 log.cc:824] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:18.345531 13438 log.cc:824] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:18.347427 13438 tablet_bootstrap.cc:492] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900: No bootstrap required, opened a new log
I20240501 19:56:18.347476 13437 tablet_bootstrap.cc:492] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6: No bootstrap required, opened a new log
I20240501 19:56:18.347615 13438 ts_tablet_manager.cc:1367] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:18.347633 13437 ts_tablet_manager.cc:1367] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 19:56:18.347662 13439 tablet_bootstrap.cc:492] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff: No bootstrap required, opened a new log
I20240501 19:56:18.347818 13439 ts_tablet_manager.cc:1367] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 19:56:18.350483 13438 raft_consensus.cc:348] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6514607627a74f67adf25a3fd2031aff" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44301 } } peers { permanent_uuid: "8f4169a210e048d9815dc6923f1d39e6" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39727 } } peers { permanent_uuid: "213f39f01d8346f2be49410119a53900" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 36717 } }
I20240501 19:56:18.350674 13438 raft_consensus.cc:374] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:18.350739 13438 raft_consensus.cc:729] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213f39f01d8346f2be49410119a53900, State: Initialized, Role: FOLLOWER
I20240501 19:56:18.350993 13439 raft_consensus.cc:348] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6514607627a74f67adf25a3fd2031aff" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44301 } } peers { permanent_uuid: "8f4169a210e048d9815dc6923f1d39e6" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39727 } } peers { permanent_uuid: "213f39f01d8346f2be49410119a53900" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 36717 } }
I20240501 19:56:18.350993 13437 raft_consensus.cc:348] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6514607627a74f67adf25a3fd2031aff" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44301 } } peers { permanent_uuid: "8f4169a210e048d9815dc6923f1d39e6" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39727 } } peers { permanent_uuid: "213f39f01d8346f2be49410119a53900" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 36717 } }
I20240501 19:56:18.351195 13437 raft_consensus.cc:374] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:18.351195 13439 raft_consensus.cc:374] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:18.351157 13438 consensus_queue.cc:260] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6514607627a74f67adf25a3fd2031aff" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44301 } } peers { permanent_uuid: "8f4169a210e048d9815dc6923f1d39e6" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39727 } } peers { permanent_uuid: "213f39f01d8346f2be49410119a53900" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 36717 } }
I20240501 19:56:18.351250 13439 raft_consensus.cc:729] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6514607627a74f67adf25a3fd2031aff, State: Initialized, Role: FOLLOWER
I20240501 19:56:18.351259 13437 raft_consensus.cc:729] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4169a210e048d9815dc6923f1d39e6, State: Initialized, Role: FOLLOWER
I20240501 19:56:18.351650 13439 consensus_queue.cc:260] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6514607627a74f67adf25a3fd2031aff" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44301 } } peers { permanent_uuid: "8f4169a210e048d9815dc6923f1d39e6" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39727 } } peers { permanent_uuid: "213f39f01d8346f2be49410119a53900" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 36717 } }
I20240501 19:56:18.351650 13437 consensus_queue.cc:260] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6514607627a74f67adf25a3fd2031aff" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44301 } } peers { permanent_uuid: "8f4169a210e048d9815dc6923f1d39e6" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39727 } } peers { permanent_uuid: "213f39f01d8346f2be49410119a53900" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 36717 } }
I20240501 19:56:18.352046 13438 ts_tablet_manager.cc:1398] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 19:56:18.352427 13437 ts_tablet_manager.cc:1398] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 19:56:18.352646 13287 heartbeater.cc:503] Master 127.8.168.126:37525 was elected leader, sending a full tablet report...
I20240501 19:56:18.352802 13154 heartbeater.cc:503] Master 127.8.168.126:37525 was elected leader, sending a full tablet report...
I20240501 19:56:18.352820 13439 ts_tablet_manager.cc:1398] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 19:56:18.353113 13420 heartbeater.cc:503] Master 127.8.168.126:37525 was elected leader, sending a full tablet report...
I20240501 19:56:18.363911 13445 raft_consensus.cc:482] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:18.364099 13445 raft_consensus.cc:504] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6514607627a74f67adf25a3fd2031aff" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44301 } } peers { permanent_uuid: "8f4169a210e048d9815dc6923f1d39e6" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39727 } } peers { permanent_uuid: "213f39f01d8346f2be49410119a53900" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 36717 } }
I20240501 19:56:18.365427 13445 leader_election.cc:288] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f4169a210e048d9815dc6923f1d39e6 (127.8.168.65:39727), 213f39f01d8346f2be49410119a53900 (127.8.168.66:36717)
I20240501 19:56:18.368477 13422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.365609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:39727 (local address 127.8.168.67:47959)
0501 19:56:18.365732 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:18.365744 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:18.365816 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:18.366364 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:18.366368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:18.366386 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:18.366605 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:18.366610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.367424 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:18.367427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:18.368107 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:18.368116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.368217 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:18.368240 (+ 23us) client_negotiation.cc:769] Sending connection context
0501 19:56:18.368286 (+ 46us) client_negotiation.cc:240] Negotiation successful
0501 19:56:18.368348 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20240501 19:56:18.368858 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.365692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:36717 (local address 127.8.168.67:56977)
0501 19:56:18.366046 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:18.366061 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:18.366168 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:18.366407 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:18.366411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:18.366430 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:18.366651 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:18.366660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.367404 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:18.367408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:18.368154 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:18.368164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.368282 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:18.368303 (+ 21us) client_negotiation.cc:769] Sending connection context
0501 19:56:18.368358 (+ 55us) client_negotiation.cc:240] Negotiation successful
0501 19:56:18.368414 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":141,"threads_started":1}
I20240501 19:56:18.369053 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.365814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:56977 (local address 127.8.168.66:36717)
0501 19:56:18.365961 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:18.365966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:18.366133 (+ 167us) server_negotiation.cc:407] Connection header received
0501 19:56:18.366261 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:18.366263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:18.366307 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:18.366382 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:18.366761 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.367272 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:18.368274 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.368764 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:18.368807 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:18.368866 (+ 59us) server_negotiation.cc:299] Negotiation successful
0501 19:56:18.368933 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20240501 19:56:18.369192 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.365960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:47959 (local address 127.8.168.65:39727)
0501 19:56:18.366104 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:18.366109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:18.366125 (+ 16us) server_negotiation.cc:407] Connection header received
0501 19:56:18.366179 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:18.366182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:18.366233 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:18.366315 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:18.366717 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.367293 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:18.368470 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.368933 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:18.368972 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:18.369024 (+ 52us) server_negotiation.cc:299] Negotiation successful
0501 19:56:18.369075 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20240501 19:56:18.369578 13241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbc63098a1a44a09eafc520aec45c29" candidate_uuid: "6514607627a74f67adf25a3fd2031aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213f39f01d8346f2be49410119a53900" is_pre_election: true
I20240501 19:56:18.369707 13108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbc63098a1a44a09eafc520aec45c29" candidate_uuid: "6514607627a74f67adf25a3fd2031aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4169a210e048d9815dc6923f1d39e6" is_pre_election: true
I20240501 19:56:18.369837 13241 raft_consensus.cc:2454] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6514607627a74f67adf25a3fd2031aff in term 0.
I20240501 19:56:18.369989 13108 raft_consensus.cc:2454] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6514607627a74f67adf25a3fd2031aff in term 0.
I20240501 19:56:18.370397 13310 leader_election.cc:302] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 213f39f01d8346f2be49410119a53900, 6514607627a74f67adf25a3fd2031aff, 8f4169a210e048d9815dc6923f1d39e6; no voters:
I20240501 19:56:18.370630 13445 raft_consensus.cc:2781] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:18.370697 13445 raft_consensus.cc:482] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:18.370741 13445 raft_consensus.cc:3037] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:18.371665 13445 raft_consensus.cc:504] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6514607627a74f67adf25a3fd2031aff" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44301 } } peers { permanent_uuid: "8f4169a210e048d9815dc6923f1d39e6" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39727 } } peers { permanent_uuid: "213f39f01d8346f2be49410119a53900" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 36717 } }
I20240501 19:56:18.371995 13445 leader_election.cc:288] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [CANDIDATE]: Term 1 election: Requested vote from peers 8f4169a210e048d9815dc6923f1d39e6 (127.8.168.65:39727), 213f39f01d8346f2be49410119a53900 (127.8.168.66:36717)
I20240501 19:56:18.372280 13108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbc63098a1a44a09eafc520aec45c29" candidate_uuid: "6514607627a74f67adf25a3fd2031aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4169a210e048d9815dc6923f1d39e6"
I20240501 19:56:18.372393 13108 raft_consensus.cc:3037] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:18.372388 13241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbc63098a1a44a09eafc520aec45c29" candidate_uuid: "6514607627a74f67adf25a3fd2031aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213f39f01d8346f2be49410119a53900"
I20240501 19:56:18.372519 13241 raft_consensus.cc:3037] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:18.373495 13108 raft_consensus.cc:2454] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6514607627a74f67adf25a3fd2031aff in term 1.
I20240501 19:56:18.373579 13241 raft_consensus.cc:2454] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6514607627a74f67adf25a3fd2031aff in term 1.
I20240501 19:56:18.373804 13308 leader_election.cc:302] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6514607627a74f67adf25a3fd2031aff, 8f4169a210e048d9815dc6923f1d39e6; no voters:
I20240501 19:56:18.373975 13445 raft_consensus.cc:2781] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:18.374184 13445 raft_consensus.cc:686] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [term 1 LEADER]: Becoming Leader. State: Replica: 6514607627a74f67adf25a3fd2031aff, State: Running, Role: LEADER
I20240501 19:56:18.374475 13445 consensus_queue.cc:237] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6514607627a74f67adf25a3fd2031aff" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44301 } } peers { permanent_uuid: "8f4169a210e048d9815dc6923f1d39e6" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39727 } } peers { permanent_uuid: "213f39f01d8346f2be49410119a53900" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 36717 } }
I20240501 19:56:18.377689 12962 catalog_manager.cc:5484] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff reported cstate change: term changed from 0 to 1, leader changed from <none> to 6514607627a74f67adf25a3fd2031aff (127.8.168.67). New cstate: current_term: 1 leader_uuid: "6514607627a74f67adf25a3fd2031aff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6514607627a74f67adf25a3fd2031aff" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f4169a210e048d9815dc6923f1d39e6" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "213f39f01d8346f2be49410119a53900" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 36717 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:18.399981 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.397075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41428 (local address 127.8.168.67:44301)
0501 19:56:18.397217 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:18.397221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:18.397234 (+ 13us) server_negotiation.cc:407] Connection header received
0501 19:56:18.397308 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:18.397310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:18.397362 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:18.397460 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:18.397792 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.398286 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:18.398898 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:18.399078 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:18.399169 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:18.399600 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:18.399710 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:18.399785 (+ 75us) server_negotiation.cc:299] Negotiation successful
0501 19:56:18.399842 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20240501 19:56:18.405511 13108 raft_consensus.cc:1261] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6 [term 1 FOLLOWER]: Refusing update from remote peer 6514607627a74f67adf25a3fd2031aff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:18.405510 13241 raft_consensus.cc:1261] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900 [term 1 FOLLOWER]: Refusing update from remote peer 6514607627a74f67adf25a3fd2031aff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:18.406163 13447 consensus_queue.cc:1035] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [LEADER]: Connected to new peer: Peer: permanent_uuid: "213f39f01d8346f2be49410119a53900" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:18.406335 13445 consensus_queue.cc:1035] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4169a210e048d9815dc6923f1d39e6" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:56:18.409198 13155 tablet.cc:2369] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:18.412340 13450 mvcc.cc:205] Tried to move back new op lower bound from 7022974477941649408 to 7022974477824495616. Current Snapshot: MvccSnapshot[applied={T|T < 7022974477941649408}]
I20240501 19:56:18.415225 13451 mvcc.cc:205] Tried to move back new op lower bound from 7022974477941649408 to 7022974477824495616. Current Snapshot: MvccSnapshot[applied={T|T < 7022974477941649408}]
I20240501 19:56:18.420459 12962 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59902:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:18.420612 12962 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59902:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:18.422953 12962 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 12d754dfa45c48c8869a00d6ce7864ce: Sending DeleteTablet for 3 replicas of tablet 4fbc63098a1a44a09eafc520aec45c29
I20240501 19:56:18.423707 13088 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fbc63098a1a44a09eafc520aec45c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:18 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:48274
I20240501 19:56:18.423731 13221 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fbc63098a1a44a09eafc520aec45c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:18 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:50946
I20240501 19:56:18.424196 13460 tablet_replica.cc:330] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900: stopping tablet replica
I20240501 19:56:18.424192 13459 tablet_replica.cc:330] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6: stopping tablet replica
I20240501 19:56:18.424301 13354 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fbc63098a1a44a09eafc520aec45c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:18 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:41414
I20240501 19:56:18.424484 13459 raft_consensus.cc:2229] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:18.424484 13460 raft_consensus.cc:2229] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:18.424825 13460 raft_consensus.cc:2258] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:18.424825 13459 raft_consensus.cc:2258] T 4fbc63098a1a44a09eafc520aec45c29 P 8f4169a210e048d9815dc6923f1d39e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:18.425431 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 13026
I20240501 19:56:18.426954 13460 ts_tablet_manager.cc:1875] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:18.429036 13461 tablet_replica.cc:330] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff: stopping tablet replica
I20240501 19:56:18.429029 13460 ts_tablet_manager.cc:1888] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:18.429122 13460 log.cc:1196] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/4fbc63098a1a44a09eafc520aec45c29
I20240501 19:56:18.429487 13460 ts_tablet_manager.cc:1909] T 4fbc63098a1a44a09eafc520aec45c29 P 213f39f01d8346f2be49410119a53900: Deleting consensus metadata
I20240501 19:56:18.429469 13461 raft_consensus.cc:2229] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:18.429878 13461 raft_consensus.cc:2258] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:18.430991 12951 catalog_manager.cc:4830] TS 213f39f01d8346f2be49410119a53900 (127.8.168.66:36717): tablet 4fbc63098a1a44a09eafc520aec45c29 (table test-table [id=b5887c187fc74d6bb438f0997ca953f2]) successfully deleted
I20240501 19:56:18.431344 13461 ts_tablet_manager.cc:1875] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:18.433444 13461 ts_tablet_manager.cc:1888] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:18.433537 13461 log.cc:1196] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/4fbc63098a1a44a09eafc520aec45c29
I20240501 19:56:18.433887 13461 ts_tablet_manager.cc:1909] T 4fbc63098a1a44a09eafc520aec45c29 P 6514607627a74f67adf25a3fd2031aff: Deleting consensus metadata
W20240501 19:56:18.434082 12949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.168.65:39727 (error 108)
I20240501 19:56:18.435089 12949 catalog_manager.cc:4830] TS 6514607627a74f67adf25a3fd2031aff (127.8.168.67:44301): tablet 4fbc63098a1a44a09eafc520aec45c29 (table test-table [id=b5887c187fc74d6bb438f0997ca953f2]) successfully deleted
I20240501 19:56:18.435137 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 13159
I20240501 19:56:18.435139 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.434829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:39727 (local address 127.0.0.1:48282)
0501 19:56:18.434968 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:18.435043 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20240501 19:56:18.435281 12949 catalog_manager.cc:4573] TS 8f4169a210e048d9815dc6923f1d39e6 (127.8.168.65:39727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4fbc63098a1a44a09eafc520aec45c29: Network error: Client connection negotiation failed: client connection to 127.8.168.65:39727: connect: Connection refused (error 111)
I20240501 19:56:18.440178 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 13292
I20240501 19:56:18.444628 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 12933
2024-05-01T19:56:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1970 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2024-05-01T19:56:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:18Z Disabled control of system clock
I20240501 19:56:18.470916 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33109
--webserver_interface=127.8.168.126
--webserver_port=0
--builtin_ntp_servers=127.8.168.84:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:33109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240501 19:56:18.576767 13466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:18.577065 13466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:18.577275 13466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:18.580605 13466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:18.580693 13466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:18.580708 13466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:18.580752 13466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:18.580775 13466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:18.584662 13466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:33109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:18.585696 13466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:18.586869 13466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:18.594259 13471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:18.594723 13474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:18.595126 13472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:18.595738 13466 server_base.cc:1034] running on GCE node
I20240501 19:56:18.596235 13466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:18.597232 13466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:18.598371 13466 hybrid_clock.cc:648] HybridClock initialized: now 1714593378598352 us; error 36 us; skew 500 ppm
I20240501 19:56:18.600327 13466 webserver.cc:460] Webserver started at http://127.8.168.126:35599/ using document root <none> and password file <none>
I20240501 19:56:18.600908 13466 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:18.600967 13466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:18.601150 13466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:18.602916 13466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "b1a491a6b3064f9b8864e966e7ed0577"
format_stamp: "Formatted at 2024-05-01 19:56:18 on dist-test-slave-tzct"
server_key: "5bcc46fdbb4af7f3a026136b837de40e"
server_key_iv: "459ca8e821db17b93221ea029f80fe0c"
server_key_version: "encryptionkey@0"
I20240501 19:56:18.603367 13466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "b1a491a6b3064f9b8864e966e7ed0577"
format_stamp: "Formatted at 2024-05-01 19:56:18 on dist-test-slave-tzct"
server_key: "5bcc46fdbb4af7f3a026136b837de40e"
server_key_iv: "459ca8e821db17b93221ea029f80fe0c"
server_key_version: "encryptionkey@0"
I20240501 19:56:18.606603 13466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:18.608798 13480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:18.609766 13466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:18.609908 13466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "b1a491a6b3064f9b8864e966e7ed0577"
format_stamp: "Formatted at 2024-05-01 19:56:18 on dist-test-slave-tzct"
server_key: "5bcc46fdbb4af7f3a026136b837de40e"
server_key_iv: "459ca8e821db17b93221ea029f80fe0c"
server_key_version: "encryptionkey@0"
I20240501 19:56:18.609995 13466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:18.614676 13466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:18.615219 13466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:18.615356 13466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:18.621646 13466 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:33109
I20240501 19:56:18.621654 13532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:33109 every 8 connection(s)
I20240501 19:56:18.622617 13466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:56:18.625137 13533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:18.625808 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 13466
I20240501 19:56:18.625954 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:56:18.626221 8865 external_mini_cluster.cc:1428] Setting key 71e66cd79160ddd98a0c3941a957ce24
I20240501 19:56:18.631263 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.627436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.8.168.126:33109)
0501 19:56:18.628011 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:18.628019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:18.628054 (+ 35us) server_negotiation.cc:407] Connection header received
0501 19:56:18.628695 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:18.628713 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:18.629049 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:18.629383 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:18.630198 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:18.630203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:18.630212 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:56:18.630221 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:18.630270 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:56:18.630282 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:18.630447 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:18.630524 (+ 77us) server_negotiation.cc:299] Negotiation successful
0501 19:56:18.630728 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
I20240501 19:56:18.632903 13533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577: Bootstrap starting.
I20240501 19:56:18.634843 13533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:18.635493 13533 log.cc:824] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:18.637174 13533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577: No bootstrap required, opened a new log
I20240501 19:56:18.639617 13533 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a491a6b3064f9b8864e966e7ed0577" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33109 } }
I20240501 19:56:18.639806 13533 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:18.639878 13533 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1a491a6b3064f9b8864e966e7ed0577, State: Initialized, Role: FOLLOWER
I20240501 19:56:18.640259 13533 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a491a6b3064f9b8864e966e7ed0577" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33109 } }
I20240501 19:56:18.640384 13533 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:18.640444 13533 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:18.640538 13533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:18.641358 13533 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a491a6b3064f9b8864e966e7ed0577" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33109 } }
I20240501 19:56:18.641678 13533 leader_election.cc:302] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1a491a6b3064f9b8864e966e7ed0577; no voters:
I20240501 19:56:18.641955 13533 leader_election.cc:288] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:18.642310 13538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:18.642515 13538 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [term 1 LEADER]: Becoming Leader. State: Replica: b1a491a6b3064f9b8864e966e7ed0577, State: Running, Role: LEADER
I20240501 19:56:18.642786 13538 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a491a6b3064f9b8864e966e7ed0577" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33109 } }
I20240501 19:56:18.642994 13533 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:18.644106 13539 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1a491a6b3064f9b8864e966e7ed0577" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a491a6b3064f9b8864e966e7ed0577" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33109 } } }
I20240501 19:56:18.644212 13539 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:18.644439 13540 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1a491a6b3064f9b8864e966e7ed0577. Latest consensus state: current_term: 1 leader_uuid: "b1a491a6b3064f9b8864e966e7ed0577" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a491a6b3064f9b8864e966e7ed0577" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33109 } } }
I20240501 19:56:18.644587 13547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:18.644673 13540 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:18.647596 13547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:18.652992 13547 catalog_manager.cc:1347] Generated new cluster ID: 7307da0c298c4522b864bb78e42c6d50
I20240501 19:56:18.653069 13547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:18.660667 13547 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:18.661060 13547 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:18.670331 13547 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577: Generated new TSK 0
I20240501 19:56:18.671139 13547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:56:18.675568 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33109
--builtin_ntp_servers=127.8.168.84:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240501 19:56:18.807727 13557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:18.807951 13557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:18.808007 13557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:18.811057 13557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:18.811115 13557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:18.811192 13557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:56:18.814913 13557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:18.815786 13557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:18.816895 13557 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:18.824879 13565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:18.825299 13562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:18.825677 13563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:18.826262 13557 server_base.cc:1034] running on GCE node
I20240501 19:56:18.826622 13557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:18.827174 13557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:18.828346 13557 hybrid_clock.cc:648] HybridClock initialized: now 1714593378828328 us; error 36 us; skew 500 ppm
I20240501 19:56:18.830268 13557 webserver.cc:460] Webserver started at http://127.8.168.65:39963/ using document root <none> and password file <none>
I20240501 19:56:18.830796 13557 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:18.830857 13557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:18.831025 13557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:18.832842 13557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "47d15b0c62784430bc148057eebdf54e"
format_stamp: "Formatted at 2024-05-01 19:56:18 on dist-test-slave-tzct"
server_key: "7c13413782381bfa39d046a42ef649a5"
server_key_iv: "8ddfa572164498d310ee95c230334d4a"
server_key_version: "encryptionkey@0"
I20240501 19:56:18.833298 13557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "47d15b0c62784430bc148057eebdf54e"
format_stamp: "Formatted at 2024-05-01 19:56:18 on dist-test-slave-tzct"
server_key: "7c13413782381bfa39d046a42ef649a5"
server_key_iv: "8ddfa572164498d310ee95c230334d4a"
server_key_version: "encryptionkey@0"
I20240501 19:56:18.836383 13557 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:18.838523 13571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:18.839485 13557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:18.839576 13557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "47d15b0c62784430bc148057eebdf54e"
format_stamp: "Formatted at 2024-05-01 19:56:18 on dist-test-slave-tzct"
server_key: "7c13413782381bfa39d046a42ef649a5"
server_key_iv: "8ddfa572164498d310ee95c230334d4a"
server_key_version: "encryptionkey@0"
I20240501 19:56:18.839650 13557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:18.844329 13557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:18.844920 13557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:18.845053 13557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:18.845521 13557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:18.846328 13557 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:18.846372 13557 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:18.846419 13557 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:18.846446 13557 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:18.854382 13557 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:38811
I20240501 19:56:18.854465 13684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:38811 every 8 connection(s)
I20240501 19:56:18.855235 13557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 19:56:18.860685 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.856887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:59299 (local address 127.8.168.126:33109)
0501 19:56:18.857013 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:18.857016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:18.857876 (+ 860us) server_negotiation.cc:407] Connection header received
0501 19:56:18.858606 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:18.858610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:18.858658 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:18.858733 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:18.860242 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:18.860246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:18.860248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:56:18.860260 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:18.860281 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:56:18.860288 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:18.860379 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:18.860504 (+ 125us) server_negotiation.cc:299] Negotiation successful
0501 19:56:18.860547 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 19:56:18.861382 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:18.857117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33109 (local address 127.8.168.65:59299)
0501 19:56:18.857694 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:18.857727 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:18.858403 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:18.858891 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:18.858899 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:18.858982 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:18.859987 (+ 1005us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 19:56:18.859989 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:18.860021 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 19:56:18.860076 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:18.860365 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:18.860372 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:18.860378 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 19:56:18.860553 (+ 175us) client_negotiation.cc:240] Negotiation successful
0501 19:56:18.860778 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":95,"threads_started":1}
I20240501 19:56:18.861570 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 13557
I20240501 19:56:18.861687 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 19:56:18.861977 8865 external_mini_cluster.cc:1428] Setting key 56396b1da81231d013fa6c8e04dc638f
I20240501 19:56:18.862640 13685 heartbeater.cc:347] Connected to a master server at 127.8.168.126:33109
I20240501 19:56:18.862851 13685 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:18.863217 13685 heartbeater.cc:511] Master 127.8.168.126:33109 requested a full tablet report, sending...
I20240501 19:56:18.863921 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33109
--builtin_ntp_servers=127.8.168.84:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20240501 19:56:18.864684 13497 ts_manager.cc:194] Registered new tserver with Master: 47d15b0c62784430bc148057eebdf54e (127.8.168.65:38811)
W20240501 19:56:18.967579 13688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:18.967804 13688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:18.967855 13688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:18.970885 13688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:18.970943 13688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:18.971022 13688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:56:18.974673 13688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:18.975616 13688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:18.976800 13688 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:18.985153 13693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:18.985229 13696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:18.985800 13694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:18.986294 13688 server_base.cc:1034] running on GCE node
I20240501 19:56:18.986701 13688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:18.987315 13688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:18.988468 13688 hybrid_clock.cc:648] HybridClock initialized: now 1714593378988452 us; error 32 us; skew 500 ppm
I20240501 19:56:18.990413 13688 webserver.cc:460] Webserver started at http://127.8.168.66:34793/ using document root <none> and password file <none>
I20240501 19:56:18.990975 13688 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:18.991036 13688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:18.991215 13688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:18.993148 13688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "3094895c49044b9ba2869bee3b38fb8d"
format_stamp: "Formatted at 2024-05-01 19:56:18 on dist-test-slave-tzct"
server_key: "87e88188cdb29083424def390214d6a4"
server_key_iv: "327b47f675b6ccf7813d8eeab3350d38"
server_key_version: "encryptionkey@0"
I20240501 19:56:18.993703 13688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "3094895c49044b9ba2869bee3b38fb8d"
format_stamp: "Formatted at 2024-05-01 19:56:18 on dist-test-slave-tzct"
server_key: "87e88188cdb29083424def390214d6a4"
server_key_iv: "327b47f675b6ccf7813d8eeab3350d38"
server_key_version: "encryptionkey@0"
I20240501 19:56:18.997140 13688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:18.999372 13702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:19.000396 13688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:19.000501 13688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "3094895c49044b9ba2869bee3b38fb8d"
format_stamp: "Formatted at 2024-05-01 19:56:18 on dist-test-slave-tzct"
server_key: "87e88188cdb29083424def390214d6a4"
server_key_iv: "327b47f675b6ccf7813d8eeab3350d38"
server_key_version: "encryptionkey@0"
I20240501 19:56:19.000595 13688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:19.005381 13688 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:19.005980 13688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:19.006120 13688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:19.006605 13688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:19.007448 13688 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:19.007495 13688 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:19.007544 13688 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:19.007572 13688 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:19.015867 13688 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:43193
I20240501 19:56:19.015892 13815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:43193 every 8 connection(s)
I20240501 19:56:19.016839 13688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 19:56:19.019318 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 13688
I20240501 19:56:19.019428 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 19:56:19.019665 8865 external_mini_cluster.cc:1428] Setting key adc2aba2e798baa96867c513283efc8e
I20240501 19:56:19.021638 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33109
--builtin_ntp_servers=127.8.168.84:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20240501 19:56:19.023013 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.018523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:33651 (local address 127.8.168.126:33109)
0501 19:56:19.018670 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:19.018675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:19.019762 (+ 1087us) server_negotiation.cc:407] Connection header received
0501 19:56:19.020614 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:19.020616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:19.020651 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:19.020741 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:19.022542 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:19.022545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:19.022548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:56:19.022558 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:19.022577 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:56:19.022584 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:19.022661 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:19.022833 (+ 172us) server_negotiation.cc:299] Negotiation successful
0501 19:56:19.022881 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 19:56:19.023834 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.018888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33109 (local address 127.8.168.66:33651)
0501 19:56:19.019610 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:19.019651 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:19.020403 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:19.020932 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:19.020943 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:19.021042 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:19.022296 (+ 1254us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 19:56:19.022297 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:19.022332 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 19:56:19.022391 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:19.022656 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:19.022663 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:19.022669 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 19:56:19.022860 (+ 191us) client_negotiation.cc:240] Negotiation successful
0501 19:56:19.023092 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
I20240501 19:56:19.024888 13816 heartbeater.cc:347] Connected to a master server at 127.8.168.126:33109
I20240501 19:56:19.025103 13816 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:19.025499 13816 heartbeater.cc:511] Master 127.8.168.126:33109 requested a full tablet report, sending...
I20240501 19:56:19.026496 13497 ts_manager.cc:194] Registered new tserver with Master: 3094895c49044b9ba2869bee3b38fb8d (127.8.168.66:43193)
W20240501 19:56:19.127705 13819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:19.127943 13819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:19.127995 13819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:19.131060 13819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:19.131115 13819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:19.131191 13819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:56:19.134914 13819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:19.135813 13819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:19.137008 13819 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:19.144651 13825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:19.145154 13827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:19.145817 13824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:19.146347 13819 server_base.cc:1034] running on GCE node
I20240501 19:56:19.146775 13819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:19.147373 13819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:19.148538 13819 hybrid_clock.cc:648] HybridClock initialized: now 1714593379148520 us; error 31 us; skew 500 ppm
I20240501 19:56:19.150544 13819 webserver.cc:460] Webserver started at http://127.8.168.67:32841/ using document root <none> and password file <none>
I20240501 19:56:19.151111 13819 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:19.151167 13819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:19.151347 13819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:19.153230 13819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "c058915ce3894b35a10db57d4caef048"
format_stamp: "Formatted at 2024-05-01 19:56:19 on dist-test-slave-tzct"
server_key: "7e16cef6bd416a6a055651d5ee23789e"
server_key_iv: "3212ca12c2bbc6b72f243d7789050486"
server_key_version: "encryptionkey@0"
I20240501 19:56:19.153697 13819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "c058915ce3894b35a10db57d4caef048"
format_stamp: "Formatted at 2024-05-01 19:56:19 on dist-test-slave-tzct"
server_key: "7e16cef6bd416a6a055651d5ee23789e"
server_key_iv: "3212ca12c2bbc6b72f243d7789050486"
server_key_version: "encryptionkey@0"
I20240501 19:56:19.156989 13819 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:19.159179 13833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:19.160185 13819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:19.160296 13819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "c058915ce3894b35a10db57d4caef048"
format_stamp: "Formatted at 2024-05-01 19:56:19 on dist-test-slave-tzct"
server_key: "7e16cef6bd416a6a055651d5ee23789e"
server_key_iv: "3212ca12c2bbc6b72f243d7789050486"
server_key_version: "encryptionkey@0"
I20240501 19:56:19.160383 13819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:19.165153 13819 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:19.165725 13819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:19.165879 13819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:19.166487 13819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:19.167446 13819 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:19.167502 13819 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:19.167562 13819 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:19.167589 13819 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:19.176828 13819 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:43903
I20240501 19:56:19.176873 13946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:43903 every 8 connection(s)
I20240501 19:56:19.177784 13819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 19:56:19.183552 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.179600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:52821 (local address 127.8.168.126:33109)
0501 19:56:19.179749 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:19.179753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:19.180695 (+ 942us) server_negotiation.cc:407] Connection header received
0501 19:56:19.181456 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:19.181459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:19.181505 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:19.181588 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:19.183177 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:19.183181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:19.183183 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:56:19.183195 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:19.183215 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:56:19.183222 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:19.183311 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:19.183406 (+ 95us) server_negotiation.cc:299] Negotiation successful
0501 19:56:19.183438 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20240501 19:56:19.184280 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.179928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33109 (local address 127.8.168.67:52821)
0501 19:56:19.180550 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:19.180584 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:19.181260 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:19.181720 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:19.181729 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:19.181842 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:19.182934 (+ 1092us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 19:56:19.182935 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:19.182966 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 19:56:19.183024 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:19.183293 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:19.183297 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:19.183301 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 19:56:19.183450 (+ 149us) client_negotiation.cc:240] Negotiation successful
0501 19:56:19.183640 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
I20240501 19:56:19.185369 13947 heartbeater.cc:347] Connected to a master server at 127.8.168.126:33109
I20240501 19:56:19.185597 13947 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:19.186053 13947 heartbeater.cc:511] Master 127.8.168.126:33109 requested a full tablet report, sending...
I20240501 19:56:19.187136 13497 ts_manager.cc:194] Registered new tserver with Master: c058915ce3894b35a10db57d4caef048 (127.8.168.67:43903)
I20240501 19:56:19.187328 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 13819
I20240501 19:56:19.187433 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 19:56:19.187671 8865 external_mini_cluster.cc:1428] Setting key 543ce4dc976b40402f7c7bffc40952b4
I20240501 19:56:19.189603 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 19:56:19.192822 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.191274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38654 (local address 127.8.168.126:33109)
0501 19:56:19.191384 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:19.191387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:19.191528 (+ 141us) server_negotiation.cc:407] Connection header received
0501 19:56:19.191630 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:19.191632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:19.191664 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:19.191735 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:19.192409 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:19.192412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:19.192415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:56:19.192427 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:19.192446 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:56:19.192454 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:19.192535 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:19.192632 (+ 97us) server_negotiation.cc:299] Negotiation successful
0501 19:56:19.192670 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240501 19:56:19.195627 13497 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:56:19.198089 13497 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:19.209964 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.207090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.8.168.67:43903)
0501 19:56:19.207436 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:19.207441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:19.207462 (+ 21us) server_negotiation.cc:407] Connection header received
0501 19:56:19.207509 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:19.207513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:19.207645 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:19.207764 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:19.208689 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:19.208696 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:19.208706 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:56:19.208742 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:19.208855 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:56:19.208872 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:19.208982 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:19.209717 (+ 735us) server_negotiation.cc:299] Negotiation successful
0501 19:56:19.209803 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":111,"threads_started":1}
I20240501 19:56:19.210322 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.206028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:43903 (local address 127.0.0.1:49760)
0501 19:56:19.206349 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:19.206391 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:19.206487 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:19.207790 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:19.207796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:19.207901 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:19.208442 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 19:56:19.208444 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:19.208488 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 19:56:19.208548 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:19.209026 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:19.209033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:19.209039 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 19:56:19.210127 (+ 1088us) client_negotiation.cc:240] Negotiation successful
0501 19:56:19.210191 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
I20240501 19:56:19.210536 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.206198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:38811 (local address 127.0.0.1:42600)
0501 19:56:19.206458 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:19.206472 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:19.206545 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:19.209174 (+ 2629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:19.209177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:19.209190 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:19.209693 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 19:56:19.209695 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:19.209706 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 19:56:19.209730 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:19.210292 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:19.210296 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:19.210299 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 19:56:19.210379 (+ 80us) client_negotiation.cc:240] Negotiation successful
0501 19:56:19.210412 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":120,"threads_started":1}
I20240501 19:56:19.210790 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.206683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:43193 (local address 127.0.0.1:60816)
0501 19:56:19.208425 (+ 1742us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:19.208438 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:19.208520 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:19.208943 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:19.208946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:19.208959 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:19.209842 (+ 883us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 19:56:19.209844 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:19.209853 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 19:56:19.209870 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:19.210605 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:19.210608 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:19.210611 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 19:56:19.210680 (+ 69us) client_negotiation.cc:240] Negotiation successful
0501 19:56:19.210708 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1676,"thread_start_us":109,"threads_started":1}
I20240501 19:56:19.211122 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.206821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60816 (local address 127.8.168.66:43193)
0501 19:56:19.207296 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:19.207302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:19.208540 (+ 1238us) server_negotiation.cc:407] Connection header received
0501 19:56:19.208610 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:19.208615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:19.208805 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:19.208950 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:19.209978 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:19.209984 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:19.209993 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:56:19.210005 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:19.210316 (+ 311us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:56:19.210335 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:19.210568 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:19.210932 (+ 364us) server_negotiation.cc:299] Negotiation successful
0501 19:56:19.210997 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":136,"threads_started":1}
I20240501 19:56:19.211311 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.206733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42600 (local address 127.8.168.65:38811)
0501 19:56:19.208778 (+ 2045us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:19.208784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:19.208805 (+ 21us) server_negotiation.cc:407] Connection header received
0501 19:56:19.208854 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:19.208857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:19.208988 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:19.209190 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:19.210034 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:19.210040 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:19.210050 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:56:19.210133 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:19.210190 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:56:19.210206 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:19.210324 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:19.210393 (+ 69us) server_negotiation.cc:299] Negotiation successful
0501 19:56:19.210449 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1929,"thread_start_us":113,"threads_started":1}
I20240501 19:56:19.213014 13881 tablet_service.cc:1467] Processing CreateTablet for tablet f10390323aa0486db8c8f3d7f2726572 (DEFAULT_TABLE table=test-table [id=cdd7b44353e7431abdfdb998571399d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:19.213591 13750 tablet_service.cc:1467] Processing CreateTablet for tablet f10390323aa0486db8c8f3d7f2726572 (DEFAULT_TABLE table=test-table [id=cdd7b44353e7431abdfdb998571399d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:19.213591 13619 tablet_service.cc:1467] Processing CreateTablet for tablet f10390323aa0486db8c8f3d7f2726572 (DEFAULT_TABLE table=test-table [id=cdd7b44353e7431abdfdb998571399d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:19.213989 13881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10390323aa0486db8c8f3d7f2726572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:19.214471 13619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10390323aa0486db8c8f3d7f2726572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:19.214471 13750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10390323aa0486db8c8f3d7f2726572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:19.220367 13964 tablet_bootstrap.cc:492] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048: Bootstrap starting.
I20240501 19:56:19.222594 13966 tablet_bootstrap.cc:492] T f10390323aa0486db8c8f3d7f2726572 P 47d15b0c62784430bc148057eebdf54e: Bootstrap starting.
I20240501 19:56:19.222594 13965 tablet_bootstrap.cc:492] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d: Bootstrap starting.
I20240501 19:56:19.222774 13964 tablet_bootstrap.cc:654] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:19.223457 13964 log.cc:824] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:19.224604 13966 tablet_bootstrap.cc:654] T f10390323aa0486db8c8f3d7f2726572 P 47d15b0c62784430bc148057eebdf54e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:19.224604 13965 tablet_bootstrap.cc:654] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:19.225091 13964 tablet_bootstrap.cc:492] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048: No bootstrap required, opened a new log
I20240501 19:56:19.225253 13964 ts_tablet_manager.cc:1367] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:19.225422 13966 log.cc:824] T f10390323aa0486db8c8f3d7f2726572 P 47d15b0c62784430bc148057eebdf54e: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:19.225422 13965 log.cc:824] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:19.227221 13966 tablet_bootstrap.cc:492] T f10390323aa0486db8c8f3d7f2726572 P 47d15b0c62784430bc148057eebdf54e: No bootstrap required, opened a new log
I20240501 19:56:19.227442 13966 ts_tablet_manager.cc:1367] T f10390323aa0486db8c8f3d7f2726572 P 47d15b0c62784430bc148057eebdf54e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240501 19:56:19.227583 13965 tablet_bootstrap.cc:492] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d: No bootstrap required, opened a new log
I20240501 19:56:19.227777 13965 ts_tablet_manager.cc:1367] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:19.228698 13964 raft_consensus.cc:348] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c058915ce3894b35a10db57d4caef048" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 43903 } } peers { permanent_uuid: "47d15b0c62784430bc148057eebdf54e" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 38811 } } peers { permanent_uuid: "3094895c49044b9ba2869bee3b38fb8d" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43193 } }
I20240501 19:56:19.228909 13964 raft_consensus.cc:374] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:19.228981 13964 raft_consensus.cc:729] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c058915ce3894b35a10db57d4caef048, State: Initialized, Role: FOLLOWER
I20240501 19:56:19.229478 13964 consensus_queue.cc:260] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c058915ce3894b35a10db57d4caef048" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 43903 } } peers { permanent_uuid: "47d15b0c62784430bc148057eebdf54e" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 38811 } } peers { permanent_uuid: "3094895c49044b9ba2869bee3b38fb8d" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43193 } }
I20240501 19:56:19.230139 13947 heartbeater.cc:503] Master 127.8.168.126:33109 was elected leader, sending a full tablet report...
I20240501 19:56:19.230312 13964 ts_tablet_manager.cc:1398] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20240501 19:56:19.231006 13970 raft_consensus.cc:482] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:19.231192 13966 raft_consensus.cc:348] T f10390323aa0486db8c8f3d7f2726572 P 47d15b0c62784430bc148057eebdf54e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c058915ce3894b35a10db57d4caef048" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 43903 } } peers { permanent_uuid: "47d15b0c62784430bc148057eebdf54e" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 38811 } } peers { permanent_uuid: "3094895c49044b9ba2869bee3b38fb8d" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43193 } }
I20240501 19:56:19.231192 13965 raft_consensus.cc:348] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c058915ce3894b35a10db57d4caef048" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 43903 } } peers { permanent_uuid: "47d15b0c62784430bc148057eebdf54e" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 38811 } } peers { permanent_uuid: "3094895c49044b9ba2869bee3b38fb8d" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43193 } }
I20240501 19:56:19.231395 13965 raft_consensus.cc:374] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:19.231364 13970 raft_consensus.cc:504] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c058915ce3894b35a10db57d4caef048" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 43903 } } peers { permanent_uuid: "47d15b0c62784430bc148057eebdf54e" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 38811 } } peers { permanent_uuid: "3094895c49044b9ba2869bee3b38fb8d" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43193 } }
I20240501 19:56:19.231469 13965 raft_consensus.cc:729] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3094895c49044b9ba2869bee3b38fb8d, State: Initialized, Role: FOLLOWER
I20240501 19:56:19.231576 13966 raft_consensus.cc:374] T f10390323aa0486db8c8f3d7f2726572 P 47d15b0c62784430bc148057eebdf54e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:19.231662 13966 raft_consensus.cc:729] T f10390323aa0486db8c8f3d7f2726572 P 47d15b0c62784430bc148057eebdf54e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d15b0c62784430bc148057eebdf54e, State: Initialized, Role: FOLLOWER
I20240501 19:56:19.231914 13965 consensus_queue.cc:260] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c058915ce3894b35a10db57d4caef048" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 43903 } } peers { permanent_uuid: "47d15b0c62784430bc148057eebdf54e" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 38811 } } peers { permanent_uuid: "3094895c49044b9ba2869bee3b38fb8d" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43193 } }
I20240501 19:56:19.232051 13966 consensus_queue.cc:260] T f10390323aa0486db8c8f3d7f2726572 P 47d15b0c62784430bc148057eebdf54e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c058915ce3894b35a10db57d4caef048" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 43903 } } peers { permanent_uuid: "47d15b0c62784430bc148057eebdf54e" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 38811 } } peers { permanent_uuid: "3094895c49044b9ba2869bee3b38fb8d" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43193 } }
I20240501 19:56:19.232975 13965 ts_tablet_manager.cc:1398] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240501 19:56:19.232990 13966 ts_tablet_manager.cc:1398] T f10390323aa0486db8c8f3d7f2726572 P 47d15b0c62784430bc148057eebdf54e: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20240501 19:56:19.233362 13685 heartbeater.cc:503] Master 127.8.168.126:33109 was elected leader, sending a full tablet report...
I20240501 19:56:19.234309 13816 heartbeater.cc:503] Master 127.8.168.126:33109 was elected leader, sending a full tablet report...
I20240501 19:56:19.234586 13970 leader_election.cc:288] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47d15b0c62784430bc148057eebdf54e (127.8.168.65:38811), 3094895c49044b9ba2869bee3b38fb8d (127.8.168.66:43193)
I20240501 19:56:19.235285 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.233091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:38811 (local address 127.8.168.67:57925)
0501 19:56:19.233377 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:19.233391 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:19.233455 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:19.234210 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:19.234214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:19.234228 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:19.234750 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 19:56:19.234752 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:19.234766 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 19:56:19.234793 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:19.235054 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:19.235057 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:19.235061 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 19:56:19.235134 (+ 73us) client_negotiation.cc:240] Negotiation successful
0501 19:56:19.235166 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20240501 19:56:19.235563 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.233588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:57925 (local address 127.8.168.65:38811)
0501 19:56:19.233711 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:19.233716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:19.233730 (+ 14us) server_negotiation.cc:407] Connection header received
0501 19:56:19.233768 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:19.233770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:19.233830 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:19.233903 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:19.234916 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:19.234920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:19.234923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:56:19.234934 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:19.234955 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:56:19.234965 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:19.235048 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:19.235412 (+ 364us) server_negotiation.cc:299] Negotiation successful
0501 19:56:19.235452 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20240501 19:56:19.236215 13639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10390323aa0486db8c8f3d7f2726572" candidate_uuid: "c058915ce3894b35a10db57d4caef048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47d15b0c62784430bc148057eebdf54e" is_pre_election: true
I20240501 19:56:19.236493 13639 raft_consensus.cc:2454] T f10390323aa0486db8c8f3d7f2726572 P 47d15b0c62784430bc148057eebdf54e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c058915ce3894b35a10db57d4caef048 in term 0.
I20240501 19:56:19.236976 13836 leader_election.cc:302] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47d15b0c62784430bc148057eebdf54e, c058915ce3894b35a10db57d4caef048; no voters:
I20240501 19:56:19.237336 13970 raft_consensus.cc:2781] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:19.237414 13970 raft_consensus.cc:482] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:19.237478 13970 raft_consensus.cc:3037] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:19.237838 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.234004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:43193 (local address 127.8.168.67:33011)
0501 19:56:19.235187 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:19.235199 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:19.235273 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:19.235607 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:19.235610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:19.235622 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:19.237220 (+ 1598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 19:56:19.237222 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:19.237234 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 19:56:19.237259 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:19.237572 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:19.237575 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:19.237578 (+ 3us) client_negotiation.cc:769] Sending connection context
0501 19:56:19.237674 (+ 96us) client_negotiation.cc:240] Negotiation successful
0501 19:56:19.237709 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"thread_start_us":81,"threads_started":1}
I20240501 19:56:19.237865 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.234527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:33011 (local address 127.8.168.66:43193)
0501 19:56:19.235262 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:19.235266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:19.235284 (+ 18us) server_negotiation.cc:407] Connection header received
0501 19:56:19.235408 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:19.235411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:19.235447 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:19.235496 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:19.237396 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:19.237399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:19.237402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:56:19.237414 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:19.237435 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:56:19.237445 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:19.237506 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:19.237693 (+ 187us) server_negotiation.cc:299] Negotiation successful
0501 19:56:19.237738 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653}
I20240501 19:56:19.238341 13770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10390323aa0486db8c8f3d7f2726572" candidate_uuid: "c058915ce3894b35a10db57d4caef048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3094895c49044b9ba2869bee3b38fb8d" is_pre_election: true
I20240501 19:56:19.238622 13770 raft_consensus.cc:2454] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c058915ce3894b35a10db57d4caef048 in term 0.
I20240501 19:56:19.239173 13970 raft_consensus.cc:504] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c058915ce3894b35a10db57d4caef048" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 43903 } } peers { permanent_uuid: "47d15b0c62784430bc148057eebdf54e" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 38811 } } peers { permanent_uuid: "3094895c49044b9ba2869bee3b38fb8d" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43193 } }
I20240501 19:56:19.239527 13970 leader_election.cc:288] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [CANDIDATE]: Term 1 election: Requested vote from peers 47d15b0c62784430bc148057eebdf54e (127.8.168.65:38811), 3094895c49044b9ba2869bee3b38fb8d (127.8.168.66:43193)
I20240501 19:56:19.239904 13639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10390323aa0486db8c8f3d7f2726572" candidate_uuid: "c058915ce3894b35a10db57d4caef048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47d15b0c62784430bc148057eebdf54e"
I20240501 19:56:19.239903 13770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10390323aa0486db8c8f3d7f2726572" candidate_uuid: "c058915ce3894b35a10db57d4caef048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3094895c49044b9ba2869bee3b38fb8d"
I20240501 19:56:19.240026 13770 raft_consensus.cc:3037] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:19.240026 13639 raft_consensus.cc:3037] T f10390323aa0486db8c8f3d7f2726572 P 47d15b0c62784430bc148057eebdf54e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:19.241232 13639 raft_consensus.cc:2454] T f10390323aa0486db8c8f3d7f2726572 P 47d15b0c62784430bc148057eebdf54e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c058915ce3894b35a10db57d4caef048 in term 1.
I20240501 19:56:19.241216 13770 raft_consensus.cc:2454] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c058915ce3894b35a10db57d4caef048 in term 1.
I20240501 19:56:19.241554 13836 leader_election.cc:302] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47d15b0c62784430bc148057eebdf54e, c058915ce3894b35a10db57d4caef048; no voters:
I20240501 19:56:19.241710 13970 raft_consensus.cc:2781] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:19.241813 13970 raft_consensus.cc:686] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [term 1 LEADER]: Becoming Leader. State: Replica: c058915ce3894b35a10db57d4caef048, State: Running, Role: LEADER
I20240501 19:56:19.242110 13970 consensus_queue.cc:237] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c058915ce3894b35a10db57d4caef048" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 43903 } } peers { permanent_uuid: "47d15b0c62784430bc148057eebdf54e" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 38811 } } peers { permanent_uuid: "3094895c49044b9ba2869bee3b38fb8d" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43193 } }
I20240501 19:56:19.245424 13496 catalog_manager.cc:5484] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 reported cstate change: term changed from 0 to 1, leader changed from <none> to c058915ce3894b35a10db57d4caef048 (127.8.168.67). New cstate: current_term: 1 leader_uuid: "c058915ce3894b35a10db57d4caef048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c058915ce3894b35a10db57d4caef048" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 43903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47d15b0c62784430bc148057eebdf54e" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3094895c49044b9ba2869bee3b38fb8d" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43193 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:19.263818 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.262233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.8.168.67:43903)
0501 19:56:19.262374 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:19.262377 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:19.262392 (+ 15us) server_negotiation.cc:407] Connection header received
0501 19:56:19.262482 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:19.262484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:19.262547 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:19.262618 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:19.263373 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:19.263376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:19.263378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:56:19.263389 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:19.263409 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:56:19.263417 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:19.263498 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:19.263596 (+ 98us) server_negotiation.cc:299] Negotiation successful
0501 19:56:19.263641 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20240501 19:56:19.267709 13817 tablet.cc:2369] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:19.269418 13770 raft_consensus.cc:1261] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d [term 1 FOLLOWER]: Refusing update from remote peer c058915ce3894b35a10db57d4caef048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:19.269418 13639 raft_consensus.cc:1261] T f10390323aa0486db8c8f3d7f2726572 P 47d15b0c62784430bc148057eebdf54e [term 1 FOLLOWER]: Refusing update from remote peer c058915ce3894b35a10db57d4caef048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:19.270062 13971 consensus_queue.cc:1035] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47d15b0c62784430bc148057eebdf54e" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:19.270226 13970 consensus_queue.cc:1035] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3094895c49044b9ba2869bee3b38fb8d" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:19.277858 13979 mvcc.cc:205] Tried to move back new op lower bound from 7022974481479892992 to 7022974481379061760. Current Snapshot: MvccSnapshot[applied={T|T < 7022974481479892992}]
I20240501 19:56:19.279165 13978 mvcc.cc:205] Tried to move back new op lower bound from 7022974481479892992 to 7022974481379061760. Current Snapshot: MvccSnapshot[applied={T|T < 7022974481479892992}]
I20240501 19:56:19.285398 13497 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38654:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:19.285578 13497 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38654:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:19.287981 13497 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b1a491a6b3064f9b8864e966e7ed0577: Sending DeleteTablet for 3 replicas of tablet f10390323aa0486db8c8f3d7f2726572
I20240501 19:56:19.288851 13750 tablet_service.cc:1514] Processing DeleteTablet for tablet f10390323aa0486db8c8f3d7f2726572 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:19 UTC) from {username='slave'} at 127.0.0.1:60816
I20240501 19:56:19.288861 13619 tablet_service.cc:1514] Processing DeleteTablet for tablet f10390323aa0486db8c8f3d7f2726572 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:19 UTC) from {username='slave'} at 127.0.0.1:42600
I20240501 19:56:19.289330 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 13557
I20240501 19:56:19.289485 13881 tablet_service.cc:1514] Processing DeleteTablet for tablet f10390323aa0486db8c8f3d7f2726572 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:19 UTC) from {username='slave'} at 127.0.0.1:49760
I20240501 19:56:19.290078 13987 tablet_replica.cc:330] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d: stopping tablet replica
I20240501 19:56:19.290084 13988 tablet_replica.cc:330] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048: stopping tablet replica
I20240501 19:56:19.290346 13988 raft_consensus.cc:2229] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:19.290406 13987 raft_consensus.cc:2229] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:19.290727 13988 raft_consensus.cc:2258] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:19.290818 13987 raft_consensus.cc:2258] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:19.292289 13987 ts_tablet_manager.cc:1875] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:19.292289 13988 ts_tablet_manager.cc:1875] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:19.294953 13988 ts_tablet_manager.cc:1888] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:19.294970 13987 ts_tablet_manager.cc:1888] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:19.295054 13987 log.cc:1196] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/f10390323aa0486db8c8f3d7f2726572
I20240501 19:56:19.295140 13988 log.cc:1196] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/f10390323aa0486db8c8f3d7f2726572
I20240501 19:56:19.295466 13987 ts_tablet_manager.cc:1909] T f10390323aa0486db8c8f3d7f2726572 P 3094895c49044b9ba2869bee3b38fb8d: Deleting consensus metadata
I20240501 19:56:19.295552 13988 ts_tablet_manager.cc:1909] T f10390323aa0486db8c8f3d7f2726572 P c058915ce3894b35a10db57d4caef048: Deleting consensus metadata
I20240501 19:56:19.296846 13481 catalog_manager.cc:4830] TS 3094895c49044b9ba2869bee3b38fb8d (127.8.168.66:43193): tablet f10390323aa0486db8c8f3d7f2726572 (table test-table [id=cdd7b44353e7431abdfdb998571399d3]) successfully deleted
I20240501 19:56:19.297061 13481 catalog_manager.cc:4830] TS c058915ce3894b35a10db57d4caef048 (127.8.168.67:43903): tablet f10390323aa0486db8c8f3d7f2726572 (table test-table [id=cdd7b44353e7431abdfdb998571399d3]) successfully deleted
W20240501 19:56:19.298388 13483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.168.65:38811 (error 108)
I20240501 19:56:19.298539 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 13688
I20240501 19:56:19.299799 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.299494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:38811 (local address 127.0.0.1:42604)
0501 19:56:19.299631 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:19.299704 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:38811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20240501 19:56:19.299966 13483 catalog_manager.cc:4573] TS 47d15b0c62784430bc148057eebdf54e (127.8.168.65:38811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f10390323aa0486db8c8f3d7f2726572: Network error: Client connection negotiation failed: client connection to 127.8.168.65:38811: connect: Connection refused (error 111)
I20240501 19:56:19.303834 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 13819
I20240501 19:56:19.308609 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 13466
2024-05-01T19:56:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (863 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20240501 19:56:19.328385 8865 mini_oidc.cc:152] Starting JWKS server
I20240501 19:56:19.329820 8865 webserver.cc:460] Webserver started at https://127.0.0.1:33547/ using document root <none> and password file <none>
I20240501 19:56:19.329977 8865 mini_oidc.cc:189] Starting OIDC Discovery server
I20240501 19:56:19.330220 8865 webserver.cc:460] Webserver started at http://127.0.0.1:36467/ using document root <none> and password file <none>
2024-05-01T19:56:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:19Z Disabled control of system clock
I20240501 19:56:19.337937 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:45387
--webserver_interface=127.8.168.126
--webserver_port=0
--builtin_ntp_servers=127.8.168.84:34559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:45387
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33547/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/testchainca.cert with env {}
W20240501 19:56:19.446010 13995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:19.446261 13995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:19.446312 13995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:19.449393 13995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:19.449463 13995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:19.449477 13995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:19.449496 13995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:19.449510 13995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240501 19:56:19.449527 13995 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240501 19:56:19.449542 13995 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33547/jwks.json
I20240501 19:56:19.454020 13995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:45387
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:45387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33547/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:19.454918 13995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:19.455739 13995 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:19.462757 14003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:19.463239 14001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:19.463629 14000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:19.464272 13995 server_base.cc:1034] running on GCE node
I20240501 19:56:19.464895 13995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:19.465903 13995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:19.467063 13995 hybrid_clock.cc:648] HybridClock initialized: now 1714593379467049 us; error 29 us; skew 500 ppm
I20240501 19:56:19.469106 13995 webserver.cc:460] Webserver started at http://127.8.168.126:39171/ using document root <none> and password file <none>
I20240501 19:56:19.469671 13995 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:19.469740 13995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:19.469951 13995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:19.471807 13995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "4a3d889f3b184b17a6c042690969f4b7"
format_stamp: "Formatted at 2024-05-01 19:56:19 on dist-test-slave-tzct"
server_key: "c3459daac2c7846b86f56654bd224aad"
server_key_iv: "a5e0b450f8be512d2d61792477059aaa"
server_key_version: "encryptionkey@0"
I20240501 19:56:19.472313 13995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "4a3d889f3b184b17a6c042690969f4b7"
format_stamp: "Formatted at 2024-05-01 19:56:19 on dist-test-slave-tzct"
server_key: "c3459daac2c7846b86f56654bd224aad"
server_key_iv: "a5e0b450f8be512d2d61792477059aaa"
server_key_version: "encryptionkey@0"
I20240501 19:56:19.475883 13995 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240501 19:56:19.478250 14009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:19.479344 13995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:19.479463 13995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "4a3d889f3b184b17a6c042690969f4b7"
format_stamp: "Formatted at 2024-05-01 19:56:19 on dist-test-slave-tzct"
server_key: "c3459daac2c7846b86f56654bd224aad"
server_key_iv: "a5e0b450f8be512d2d61792477059aaa"
server_key_version: "encryptionkey@0"
I20240501 19:56:19.479565 13995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:19.504221 13995 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:19.504977 13995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:19.505142 13995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:19.511708 13995 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:45387
I20240501 19:56:19.511719 14061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:45387 every 8 connection(s)
I20240501 19:56:19.512748 13995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:56:19.513201 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 13995
I20240501 19:56:19.513320 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:56:19.513585 8865 external_mini_cluster.cc:1428] Setting key e96fb780e8edae41acdf4c7e97086087
I20240501 19:56:19.515689 14062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:19.520742 14062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7: Bootstrap starting.
I20240501 19:56:19.521642 14065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.514779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.8.168.126:45387)
0501 19:56:19.515391 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:19.515400 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:19.515434 (+ 34us) server_negotiation.cc:407] Connection header received
0501 19:56:19.516480 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:19.516500 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:19.516900 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:19.517239 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:19.517728 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:19.518504 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:19.519186 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:19.519456 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:19.520068 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:19.520129 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:19.520140 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:56:19.520497 (+ 357us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:19.520541 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:56:19.520556 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:19.520659 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:19.520838 (+ 179us) server_negotiation.cc:299] Negotiation successful
0501 19:56:19.521068 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":133,"threads_started":1}
I20240501 19:56:19.522866 14062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:19.523715 14062 log.cc:824] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:19.524446 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 19:56:19.525759 14062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7: No bootstrap required, opened a new log
I20240501 19:56:19.528803 14062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3d889f3b184b17a6c042690969f4b7" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45387 } }
I20240501 19:56:19.528998 14062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:19.529064 14062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a3d889f3b184b17a6c042690969f4b7, State: Initialized, Role: FOLLOWER
I20240501 19:56:19.529480 14062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3d889f3b184b17a6c042690969f4b7" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45387 } }
I20240501 19:56:19.529604 14062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:19.529662 14062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:19.529762 14062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:19.530568 14062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3d889f3b184b17a6c042690969f4b7" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45387 } }
I20240501 19:56:19.530882 14062 leader_election.cc:302] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a3d889f3b184b17a6c042690969f4b7; no voters:
I20240501 19:56:19.531142 14062 leader_election.cc:288] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:19.531261 14067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:19.531523 14067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [term 1 LEADER]: Becoming Leader. State: Replica: 4a3d889f3b184b17a6c042690969f4b7, State: Running, Role: LEADER
I20240501 19:56:19.531939 14067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3d889f3b184b17a6c042690969f4b7" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45387 } }
I20240501 19:56:19.532328 14062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:19.533601 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a3d889f3b184b17a6c042690969f4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3d889f3b184b17a6c042690969f4b7" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45387 } } }
I20240501 19:56:19.533700 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:19.533928 14069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a3d889f3b184b17a6c042690969f4b7. Latest consensus state: current_term: 1 leader_uuid: "4a3d889f3b184b17a6c042690969f4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3d889f3b184b17a6c042690969f4b7" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45387 } } }
I20240501 19:56:19.534027 14076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:19.534093 14069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:19.537118 14076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:19.542866 14076 catalog_manager.cc:1347] Generated new cluster ID: 3582a01782ae40698f5197eade99fd0c
I20240501 19:56:19.542953 14076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:19.578099 14076 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:19.579068 14076 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:19.586247 14076 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4a3d889f3b184b17a6c042690969f4b7: Generated new TSK 0
I20240501 19:56:19.587134 14076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:56:19.623070 14065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.598351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.8.168.126:45387)
0501 19:56:19.598489 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:19.598493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:19.598609 (+ 116us) server_negotiation.cc:407] Connection header received
0501 19:56:19.598836 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:19.598840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:19.598905 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:19.599033 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0501 19:56:19.599368 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:19.600198 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:19.601141 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:19.601352 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:19.601468 (+ 116us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0501 19:56:19.622538 (+ 21070us) server_negotiation.cc:377] Sending JWT_EXCHANGE NegotiatePB response
0501 19:56:19.622666 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:19.622824 (+ 158us) server_negotiation.cc:299] Negotiation successful
0501 19:56:19.622894 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55,"thread_start_us":186,"threads_started":1}
I20240501 19:56:19.639241 14065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:19.635361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38604 (local address 127.8.168.126:45387)
0501 19:56:19.635495 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:19.635498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:19.635561 (+ 63us) server_negotiation.cc:407] Connection header received
0501 19:56:19.635712 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:19.635716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:19.635769 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:19.635851 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0501 19:56:19.636162 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:19.636986 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:19.637602 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:19.637794 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:19.637847 (+ 53us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0501 19:56:19.638872 (+ 1025us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0501 19:56:19.639104 (+ 232us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38604: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":53}
W20240501 19:56:19.639329 14065 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38604: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20240501 19:56:19.639917 14098 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:19.635253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45387 (local address 127.0.0.1:38604)
0501 19:56:19.635498 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:19.635511 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:19.635613 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:19.635867 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:19.635869 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:19.635875 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0501 19:56:19.636072 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:19.636078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:19.637083 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:19.637085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:19.637503 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:19.637511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:19.637599 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:19.637620 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0501 19:56:19.639058 (+ 1438us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0501 19:56:19.639207 (+ 149us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.168.126:45387: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
I20240501 19:56:34.647261 14104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:34.644079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34598 (local address 127.8.168.126:45387)
0501 19:56:34.644339 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:34.644343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:34.644360 (+ 17us) server_negotiation.cc:407] Connection header received
0501 19:56:34.644415 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:34.644418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:34.644464 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:34.644568 (+ 104us) server_negotiation.cc:226] Negotiated authn=JWT
0501 19:56:34.644982 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:34.645498 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:34.646194 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:34.646325 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:34.646363 (+ 38us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0501 19:56:34.646985 (+ 622us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0501 19:56:34.647148 (+ 163us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34598: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":87,"threads_started":1}
W20240501 19:56:34.647325 14104 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34598: JWT verification failed: token expired
W20240501 19:56:34.647367 14103 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:34.643956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45387 (local address 127.0.0.1:34598)
0501 19:56:34.644233 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:34.644247 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:34.644340 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:34.644630 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:34.644633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:34.644641 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0501 19:56:34.644864 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:34.644872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:34.645631 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:34.645634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:34.646101 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:34.646110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:34.646209 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:34.646228 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0501 19:56:34.647117 (+ 889us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0501 19:56:34.647234 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.168.126:45387: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":106,"threads_started":1}
W20240501 19:56:34.649646 14104 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:34.649034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34604 (local address 127.8.168.126:45387)
0501 19:56:34.649169 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:34.649172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:34.649186 (+ 14us) server_negotiation.cc:407] Connection header received
0501 19:56:34.649278 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:34.649281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:34.649319 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:34.649385 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:34.649540 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34604: BlockingRecv error: recv got EOF from 127.0.0.1:34604 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20240501 19:56:34.652963 14114 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.8.168.126:45387
W20240501 19:56:34.653471 14104 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:34.652661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.8.168.126:45387)
0501 19:56:34.652814 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:34.652817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:34.652899 (+ 82us) server_negotiation.cc:407] Connection header received
0501 19:56:34.653112 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:34.653115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:34.653160 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:34.653228 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:34.653367 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34616: BlockingRecv error: recv got EOF from 127.0.0.1:34616 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20240501 19:56:34.653635 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 13995
2024-05-01T19:56:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15364 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20240501 19:56:34.692204 8865 mini_oidc.cc:152] Starting JWKS server
I20240501 19:56:34.693229 8865 webserver.cc:460] Webserver started at https://127.0.0.1:38013/ using document root <none> and password file <none>
I20240501 19:56:34.693327 8865 mini_oidc.cc:189] Starting OIDC Discovery server
I20240501 19:56:34.693559 8865 webserver.cc:460] Webserver started at http://127.0.0.1:36511/ using document root <none> and password file <none>
2024-05-01T19:56:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:34Z Disabled control of system clock
I20240501 19:56:34.700677 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:38253
--webserver_interface=127.8.168.126
--webserver_port=0
--builtin_ntp_servers=127.8.168.84:46377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:38253
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38013/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/testchainca.cert with env {}
W20240501 19:56:34.805444 14120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:34.805683 14120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:34.805734 14120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:34.808815 14120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:34.808888 14120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:34.808945 14120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:34.808980 14120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:34.809005 14120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240501 19:56:34.809052 14120 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240501 19:56:34.809073 14120 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38013/jwks.json
I20240501 19:56:34.813498 14120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:46377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:38253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:38253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38013/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:34.814422 14120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:34.815268 14120 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:56:34.822798 14120 server_base.cc:1034] running on GCE node
W20240501 19:56:34.823143 14126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:34.823700 14125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:34.823945 14128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:34.824514 14120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:34.825466 14120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:34.826617 14120 hybrid_clock.cc:648] HybridClock initialized: now 1714593394826601 us; error 37 us; skew 500 ppm
I20240501 19:56:34.828624 14120 webserver.cc:460] Webserver started at http://127.8.168.126:33079/ using document root <none> and password file <none>
I20240501 19:56:34.829208 14120 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:34.829262 14120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:34.829439 14120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:34.831209 14120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "a572c4db46574457b4b9231f155a7a7c"
format_stamp: "Formatted at 2024-05-01 19:56:34 on dist-test-slave-tzct"
server_key: "6502e126772e621ddea285cf1ea27d84"
server_key_iv: "1aec4f1fa2f81b52d4b00557900ec25c"
server_key_version: "encryptionkey@0"
I20240501 19:56:34.831688 14120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "a572c4db46574457b4b9231f155a7a7c"
format_stamp: "Formatted at 2024-05-01 19:56:34 on dist-test-slave-tzct"
server_key: "6502e126772e621ddea285cf1ea27d84"
server_key_iv: "1aec4f1fa2f81b52d4b00557900ec25c"
server_key_version: "encryptionkey@0"
I20240501 19:56:34.835040 14120 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:34.837260 14134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:34.838270 14120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:34.838384 14120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "a572c4db46574457b4b9231f155a7a7c"
format_stamp: "Formatted at 2024-05-01 19:56:34 on dist-test-slave-tzct"
server_key: "6502e126772e621ddea285cf1ea27d84"
server_key_iv: "1aec4f1fa2f81b52d4b00557900ec25c"
server_key_version: "encryptionkey@0"
I20240501 19:56:34.838498 14120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:34.860795 14120 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:34.861552 14120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:34.861723 14120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:34.868454 14120 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:38253
I20240501 19:56:34.868459 14186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:38253 every 8 connection(s)
I20240501 19:56:34.869495 14120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:56:34.872342 14187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:34.875248 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 14120
I20240501 19:56:34.875368 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:56:34.875622 8865 external_mini_cluster.cc:1428] Setting key 4f28cb0c5d044837f488afe5348857ae
I20240501 19:56:34.878415 14187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c: Bootstrap starting.
I20240501 19:56:34.880401 14187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:34.881155 14187 log.cc:824] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:34.882927 14187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c: No bootstrap required, opened a new log
I20240501 19:56:34.883098 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:34.876847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46764 (local address 127.8.168.126:38253)
0501 19:56:34.877433 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:34.877442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:34.877475 (+ 33us) server_negotiation.cc:407] Connection header received
0501 19:56:34.878090 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:34.878110 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:34.878448 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:34.878790 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:34.879254 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:34.879991 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:34.880671 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:34.880961 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:34.881508 (+ 547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:34.881530 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:34.881543 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:56:34.882026 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:34.882069 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:56:34.882086 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:34.882180 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:34.882315 (+ 135us) server_negotiation.cc:299] Negotiation successful
0501 19:56:34.882529 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":110,"threads_started":1}
I20240501 19:56:34.885574 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 19:56:34.885953 14187 raft_consensus.cc:348] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a572c4db46574457b4b9231f155a7a7c" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 38253 } }
I20240501 19:56:34.886147 14187 raft_consensus.cc:374] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:34.886204 14187 raft_consensus.cc:729] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a572c4db46574457b4b9231f155a7a7c, State: Initialized, Role: FOLLOWER
I20240501 19:56:34.886600 14187 consensus_queue.cc:260] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a572c4db46574457b4b9231f155a7a7c" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 38253 } }
I20240501 19:56:34.886720 14187 raft_consensus.cc:388] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:34.886775 14187 raft_consensus.cc:482] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:34.886878 14187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:34.887708 14187 raft_consensus.cc:504] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a572c4db46574457b4b9231f155a7a7c" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 38253 } }
I20240501 19:56:34.888025 14187 leader_election.cc:302] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a572c4db46574457b4b9231f155a7a7c; no voters:
I20240501 19:56:34.888335 14187 leader_election.cc:288] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:34.888442 14192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:34.888679 14192 raft_consensus.cc:686] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [term 1 LEADER]: Becoming Leader. State: Replica: a572c4db46574457b4b9231f155a7a7c, State: Running, Role: LEADER
I20240501 19:56:34.889057 14192 consensus_queue.cc:237] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a572c4db46574457b4b9231f155a7a7c" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 38253 } }
I20240501 19:56:34.889467 14187 sys_catalog.cc:564] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:34.890617 14193 sys_catalog.cc:455] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a572c4db46574457b4b9231f155a7a7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a572c4db46574457b4b9231f155a7a7c" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 38253 } } }
I20240501 19:56:34.890599 14194 sys_catalog.cc:455] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a572c4db46574457b4b9231f155a7a7c. Latest consensus state: current_term: 1 leader_uuid: "a572c4db46574457b4b9231f155a7a7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a572c4db46574457b4b9231f155a7a7c" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 38253 } } }
I20240501 19:56:34.890774 14193 sys_catalog.cc:458] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:34.890776 14194 sys_catalog.cc:458] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:34.891649 14202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:34.894197 14202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:34.898753 14202 catalog_manager.cc:1347] Generated new cluster ID: d74114d74dc944d1bdcab45d86ab12c4
I20240501 19:56:34.898828 14202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:34.920485 14202 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:34.921824 14202 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:34.934338 14202 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a572c4db46574457b4b9231f155a7a7c: Generated new TSK 0
I20240501 19:56:34.935081 14202 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20240501 19:56:34.960391 14217 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20240501 19:56:34.960469 14217 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20240501 19:56:34.961313 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:34.945525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46778 (local address 127.8.168.126:38253)
0501 19:56:34.945637 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:34.945640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:34.945752 (+ 112us) server_negotiation.cc:407] Connection header received
0501 19:56:34.945912 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:34.945916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:34.945976 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:34.946103 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0501 19:56:34.946422 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:34.947237 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:34.947859 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:34.948064 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:34.948165 (+ 101us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0501 19:56:34.960864 (+ 12699us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0501 19:56:34.961162 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46778: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":33}
W20240501 19:56:34.961366 14216 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:34.945418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:38253 (local address 127.0.0.1:46778)
0501 19:56:34.945665 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:34.945677 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:34.945770 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:34.946109 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:34.946111 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:34.946117 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0501 19:56:34.946328 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:34.946333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:34.947338 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:34.947340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:34.947754 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:34.947762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:34.947847 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:34.947867 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0501 19:56:34.961082 (+ 13215us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0501 19:56:34.961226 (+ 144us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.168.126:38253: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":93,"threads_started":1}
W20240501 19:56:34.961411 14190 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46778: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20240501 19:56:34.962010 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 14120
2024-05-01T19:56:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (307 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20240501 19:56:34.999769 8865 mini_oidc.cc:152] Starting JWKS server
I20240501 19:56:35.000645 8865 webserver.cc:460] Webserver started at https://127.0.0.1:34991/ using document root <none> and password file <none>
I20240501 19:56:35.000763 8865 mini_oidc.cc:189] Starting OIDC Discovery server
I20240501 19:56:35.001008 8865 webserver.cc:460] Webserver started at http://127.0.0.1:43621/ using document root <none> and password file <none>
2024-05-01T19:56:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:35Z Disabled control of system clock
I20240501 19:56:35.008256 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:36431
--webserver_interface=127.8.168.126
--webserver_port=0
--builtin_ntp_servers=127.8.168.84:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:36431
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34991/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20240501 19:56:35.111500 14223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:35.111723 14223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:35.111769 14223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:35.114763 14223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:35.114820 14223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:35.114835 14223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:35.114851 14223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:35.114866 14223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240501 19:56:35.114881 14223 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240501 19:56:35.114907 14223 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34991/jwks.json
I20240501 19:56:35.118427 14223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:36431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:36431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34991/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:35.119230 14223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:35.120254 14223 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:35.127493 14231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:35.127569 14229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:35.127995 14228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:35.128563 14223 server_base.cc:1034] running on GCE node
I20240501 19:56:35.129042 14223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:35.129855 14223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:35.130993 14223 hybrid_clock.cc:648] HybridClock initialized: now 1714593395130985 us; error 25 us; skew 500 ppm
I20240501 19:56:35.137320 14223 webserver.cc:460] Webserver started at http://127.8.168.126:41603/ using document root <none> and password file <none>
I20240501 19:56:35.137810 14223 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:35.137858 14223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:35.138010 14223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:35.139864 14223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "738f626c131c478e98ec26fba86b9279"
format_stamp: "Formatted at 2024-05-01 19:56:35 on dist-test-slave-tzct"
server_key: "3bb4a2819c22e97058d109a6fa948e07"
server_key_iv: "7fb0b3fd9324253a787383496410e833"
server_key_version: "encryptionkey@0"
I20240501 19:56:35.140291 14223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "738f626c131c478e98ec26fba86b9279"
format_stamp: "Formatted at 2024-05-01 19:56:35 on dist-test-slave-tzct"
server_key: "3bb4a2819c22e97058d109a6fa948e07"
server_key_iv: "7fb0b3fd9324253a787383496410e833"
server_key_version: "encryptionkey@0"
I20240501 19:56:35.143404 14223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:35.145488 14237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:35.146481 14223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:35.146582 14223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "738f626c131c478e98ec26fba86b9279"
format_stamp: "Formatted at 2024-05-01 19:56:35 on dist-test-slave-tzct"
server_key: "3bb4a2819c22e97058d109a6fa948e07"
server_key_iv: "7fb0b3fd9324253a787383496410e833"
server_key_version: "encryptionkey@0"
I20240501 19:56:35.146664 14223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:35.171342 14223 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:35.171973 14223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:35.172114 14223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:35.178958 14289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:36431 every 8 connection(s)
I20240501 19:56:35.178983 14223 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:36431
I20240501 19:56:35.179903 14223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:56:35.182309 14290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.183094 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 14223
I20240501 19:56:35.183204 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:56:35.183463 8865 external_mini_cluster.cc:1428] Setting key 119e88abb608c35a72fb238cd0bea42d
I20240501 19:56:35.187826 14290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279: Bootstrap starting.
I20240501 19:56:35.189831 14290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.190526 14290 log.cc:824] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:35.190568 14293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:35.184653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49454 (local address 127.8.168.126:36431)
0501 19:56:35.185256 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:35.185265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:35.185297 (+ 32us) server_negotiation.cc:407] Connection header received
0501 19:56:35.185844 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:35.185860 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:35.186171 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:35.186479 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:35.186946 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:35.187688 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:35.188360 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:35.188621 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:35.189141 (+ 520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:35.189181 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:35.189192 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:56:35.189524 (+ 332us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:35.189576 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:56:35.189591 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:35.189682 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:35.189815 (+ 133us) server_negotiation.cc:299] Negotiation successful
0501 19:56:35.190015 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":120,"threads_started":1}
I20240501 19:56:35.192250 14290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279: No bootstrap required, opened a new log
I20240501 19:56:35.193154 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 19:56:35.195228 14290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f626c131c478e98ec26fba86b9279" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 36431 } }
I20240501 19:56:35.195415 14290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.195472 14290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 738f626c131c478e98ec26fba86b9279, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.195847 14290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f626c131c478e98ec26fba86b9279" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 36431 } }
I20240501 19:56:35.195979 14290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:35.196034 14290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:35.196131 14290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.196871 14290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f626c131c478e98ec26fba86b9279" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 36431 } }
I20240501 19:56:35.197167 14290 leader_election.cc:302] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 738f626c131c478e98ec26fba86b9279; no voters:
I20240501 19:56:35.197402 14290 leader_election.cc:288] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:35.197494 14295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:35.197719 14295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [term 1 LEADER]: Becoming Leader. State: Replica: 738f626c131c478e98ec26fba86b9279, State: Running, Role: LEADER
I20240501 19:56:35.197993 14295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f626c131c478e98ec26fba86b9279" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 36431 } }
I20240501 19:56:35.198340 14290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:35.199890 14296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "738f626c131c478e98ec26fba86b9279" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f626c131c478e98ec26fba86b9279" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 36431 } } }
I20240501 19:56:35.199927 14297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 738f626c131c478e98ec26fba86b9279. Latest consensus state: current_term: 1 leader_uuid: "738f626c131c478e98ec26fba86b9279" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "738f626c131c478e98ec26fba86b9279" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 36431 } } }
I20240501 19:56:35.200021 14296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:35.200021 14297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279 [sys.catalog]: This master's current role is: LEADER
W20240501 19:56:35.202472 14310 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:56:35.202543 14310 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:56:35.202631 14309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:35.203425 14309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:35.208078 14309 catalog_manager.cc:1347] Generated new cluster ID: b1d351845c3240f8a594ede8e06f4665
I20240501 19:56:35.208140 14309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:35.245880 14309 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:35.246776 14309 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:35.259471 14309 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 738f626c131c478e98ec26fba86b9279: Generated new TSK 0
I20240501 19:56:35.260123 14309 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20240501 19:56:35.298053 14320 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20240501 19:56:35.298148 14320 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20240501 19:56:35.299465 14293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:35.276654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.8.168.126:36431)
0501 19:56:35.276798 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:35.276802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:35.276929 (+ 127us) server_negotiation.cc:407] Connection header received
0501 19:56:35.277094 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:35.277097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:35.277143 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:35.277240 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0501 19:56:35.277584 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:35.278084 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:35.278706 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:35.278875 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:35.278957 (+ 82us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0501 19:56:35.298961 (+ 20004us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34991/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0501 19:56:35.299347 (+ 386us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49462: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34991/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":36}
W20240501 19:56:35.299484 14319 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:35.276573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:36431 (local address 127.0.0.1:49462)
0501 19:56:35.276856 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:35.276868 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:35.276967 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:35.277249 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:35.277252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:35.277260 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0501 19:56:35.277484 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:35.277490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:35.278190 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:35.278194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:35.278605 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:35.278613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:35.278713 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:35.278736 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0501 19:56:35.299157 (+ 20421us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34991/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0501 19:56:35.299292 (+ 135us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.168.126:36431: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34991/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":104,"threads_started":1}
W20240501 19:56:35.299535 14293 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49462: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34991/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20240501 19:56:35.300251 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 14223
2024-05-01T19:56:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (338 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2024-05-01T19:56:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:35Z Disabled control of system clock
I20240501 19:56:35.344923 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:41071
--webserver_interface=127.8.168.126
--webserver_port=0
--builtin_ntp_servers=127.8.168.84:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:41071
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/testchainca.cert with env {}
W20240501 19:56:35.446017 14324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:35.446233 14324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:35.446300 14324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:35.449913 14324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:35.449985 14324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:35.450007 14324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:35.450032 14324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:35.450064 14324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240501 19:56:35.450089 14324 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240501 19:56:35.450116 14324 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20240501 19:56:35.454319 14324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:41071
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:41071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:35.455195 14324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:35.455996 14324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:35.462574 14332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:35.463176 14330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:35.463578 14329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:35.464223 14324 server_base.cc:1034] running on GCE node
I20240501 19:56:35.464666 14324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:35.465564 14324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:35.466710 14324 hybrid_clock.cc:648] HybridClock initialized: now 1714593395466699 us; error 27 us; skew 500 ppm
I20240501 19:56:35.468410 14324 webserver.cc:460] Webserver started at http://127.8.168.126:39443/ using document root <none> and password file <none>
I20240501 19:56:35.468951 14324 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:35.469013 14324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:35.469180 14324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:35.470867 14324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "66e2d3b34eaa4e23bf775639a10633ec"
format_stamp: "Formatted at 2024-05-01 19:56:35 on dist-test-slave-tzct"
server_key: "1e5716c82a5e513fc483b78250e302b3"
server_key_iv: "a41e5d38d831b22f44f449661bdbca54"
server_key_version: "encryptionkey@0"
I20240501 19:56:35.471338 14324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "66e2d3b34eaa4e23bf775639a10633ec"
format_stamp: "Formatted at 2024-05-01 19:56:35 on dist-test-slave-tzct"
server_key: "1e5716c82a5e513fc483b78250e302b3"
server_key_iv: "a41e5d38d831b22f44f449661bdbca54"
server_key_version: "encryptionkey@0"
I20240501 19:56:35.474519 14324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:35.476573 14338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:35.477540 14324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:35.477646 14324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "66e2d3b34eaa4e23bf775639a10633ec"
format_stamp: "Formatted at 2024-05-01 19:56:35 on dist-test-slave-tzct"
server_key: "1e5716c82a5e513fc483b78250e302b3"
server_key_iv: "a41e5d38d831b22f44f449661bdbca54"
server_key_version: "encryptionkey@0"
I20240501 19:56:35.477733 14324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:35.489409 14324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:35.490006 14324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:35.490162 14324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:35.496544 14324 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:41071
I20240501 19:56:35.496592 14390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:41071 every 8 connection(s)
I20240501 19:56:35.497520 14324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:56:35.499099 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 14324
I20240501 19:56:35.499207 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:56:35.499473 8865 external_mini_cluster.cc:1428] Setting key 347d3ce200747b15eea99da87ac92899
I20240501 19:56:35.500332 14391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.505630 14391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec: Bootstrap starting.
I20240501 19:56:35.506836 14394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:35.500587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57044 (local address 127.8.168.126:41071)
0501 19:56:35.501185 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:35.501193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:35.501226 (+ 33us) server_negotiation.cc:407] Connection header received
0501 19:56:35.502178 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:35.502197 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:35.502515 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:35.502830 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:35.503306 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:35.504008 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:35.504653 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:35.504925 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:35.505406 (+ 481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:35.505459 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:35.505469 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:56:35.505830 (+ 361us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:35.505870 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:56:35.505886 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:35.505977 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:35.506111 (+ 134us) server_negotiation.cc:299] Negotiation successful
0501 19:56:35.506321 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":137,"threads_started":1}
I20240501 19:56:35.507594 14391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:35.508430 14391 log.cc:824] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:35.509349 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 19:56:35.510267 14391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec: No bootstrap required, opened a new log
I20240501 19:56:35.513319 14391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e2d3b34eaa4e23bf775639a10633ec" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41071 } }
I20240501 19:56:35.513512 14391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:35.513557 14391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2d3b34eaa4e23bf775639a10633ec, State: Initialized, Role: FOLLOWER
I20240501 19:56:35.513906 14391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e2d3b34eaa4e23bf775639a10633ec" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41071 } }
I20240501 19:56:35.514017 14391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:35.514082 14391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:35.514190 14391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:35.515162 14391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e2d3b34eaa4e23bf775639a10633ec" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41071 } }
I20240501 19:56:35.515488 14391 leader_election.cc:302] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66e2d3b34eaa4e23bf775639a10633ec; no voters:
I20240501 19:56:35.515751 14391 leader_election.cc:288] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:35.515841 14396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:35.516059 14396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [term 1 LEADER]: Becoming Leader. State: Replica: 66e2d3b34eaa4e23bf775639a10633ec, State: Running, Role: LEADER
I20240501 19:56:35.516321 14396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e2d3b34eaa4e23bf775639a10633ec" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41071 } }
I20240501 19:56:35.516809 14391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:35.517807 14397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66e2d3b34eaa4e23bf775639a10633ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e2d3b34eaa4e23bf775639a10633ec" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41071 } } }
I20240501 19:56:35.517968 14397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:35.518333 14405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:35.518857 14398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66e2d3b34eaa4e23bf775639a10633ec. Latest consensus state: current_term: 1 leader_uuid: "66e2d3b34eaa4e23bf775639a10633ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e2d3b34eaa4e23bf775639a10633ec" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41071 } } }
I20240501 19:56:35.518970 14398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:35.521315 14405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:35.526577 14405 catalog_manager.cc:1347] Generated new cluster ID: 6dda9e6b2c304f56a0b76466bd02ef88
I20240501 19:56:35.526644 14405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:35.538949 14405 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:35.539887 14405 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:35.551069 14405 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 66e2d3b34eaa4e23bf775639a10633ec: Generated new TSK 0
I20240501 19:56:35.551762 14405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:56:35.609416 14394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:35.564219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57060 (local address 127.8.168.126:41071)
0501 19:56:35.564346 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:35.564349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:35.564456 (+ 107us) server_negotiation.cc:407] Connection header received
0501 19:56:35.564622 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:35.564625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:35.564671 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:35.564796 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0501 19:56:35.565230 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:35.565749 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:35.566616 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:35.566751 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:35.566831 (+ 80us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0501 19:56:35.609045 (+ 42214us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0501 19:56:35.609295 (+ 250us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57060: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":51}
W20240501 19:56:35.609493 14394 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57060: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20240501 19:56:35.609625 14420 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:35.564081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41071 (local address 127.0.0.1:57060)
0501 19:56:35.564369 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:35.564384 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:35.564496 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:35.564811 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:35.564815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:35.564823 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0501 19:56:35.565124 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:35.565132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:35.565845 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:35.565849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:35.566518 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:35.566529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:35.566644 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:35.566668 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0501 19:56:35.609260 (+ 42592us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0501 19:56:35.609443 (+ 183us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.168.126:41071: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":113,"threads_started":1}
I20240501 19:56:35.610256 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 14324
2024-05-01T19:56:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (289 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (111 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (112 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 19:56:35 dist-test-slave-tzct krb5kdc[14432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 19:56:35 dist-test-slave-tzct krb5kdc[14432](info): set up 2 sockets
May 01 19:56:35 dist-test-slave-tzct krb5kdc[14432](info): commencing operation
krb5kdc: starting...
W20240501 19:56:36.721249 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.869s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 19:56:36 dist-test-slave-tzct krb5kdc[14432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T19:56:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 19:56:36.834151 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33287
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:33287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:36.939816 14445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:36.940081 14445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:36.940155 14445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:36.943336 14445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:36.943413 14445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:36.943445 14445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:36.943468 14445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:36.943497 14445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:36.947815 14445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:33287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:36.948765 14445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:36.949628 14445 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:36.956619 14453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:36.957207 14450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:36.957587 14451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:36.958283 14445 server_base.cc:1034] running on GCE node
I20240501 19:56:36.958827 14445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:36.959806 14445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:36.960956 14445 hybrid_clock.cc:648] HybridClock initialized: now 1714593396960935 us; error 38 us; skew 500 ppm
May 01 19:56:36 dist-test-slave-tzct krb5kdc[14432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593396, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:36.963691 14445 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 19:56:36.964684 14445 webserver.cc:460] Webserver started at http://127.8.168.126:39903/ using document root <none> and password file <none>
I20240501 19:56:36.965250 14445 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:36.965312 14445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:36.965493 14445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:36.967301 14445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "ca1f5d0597de4dc6b069bb6245aef501"
format_stamp: "Formatted at 2024-05-01 19:56:36 on dist-test-slave-tzct"
server_key: "6ed7dc89abf0be76664a78dd64000c8e"
server_key_iv: "d9934da35e5462f80616e3d830f39ee5"
server_key_version: "encryptionkey@0"
I20240501 19:56:36.967795 14445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "ca1f5d0597de4dc6b069bb6245aef501"
format_stamp: "Formatted at 2024-05-01 19:56:36 on dist-test-slave-tzct"
server_key: "6ed7dc89abf0be76664a78dd64000c8e"
server_key_iv: "d9934da35e5462f80616e3d830f39ee5"
server_key_version: "encryptionkey@0"
I20240501 19:56:36.971231 14445 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20240501 19:56:36.973575 14460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:36.974623 14445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240501 19:56:36.974752 14445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "ca1f5d0597de4dc6b069bb6245aef501"
format_stamp: "Formatted at 2024-05-01 19:56:36 on dist-test-slave-tzct"
server_key: "6ed7dc89abf0be76664a78dd64000c8e"
server_key_iv: "d9934da35e5462f80616e3d830f39ee5"
server_key_version: "encryptionkey@0"
I20240501 19:56:36.974857 14445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:36.997655 14445 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:37.000558 14445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:37.000771 14445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:37.007340 14445 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:33287
I20240501 19:56:37.007431 14512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:33287 every 8 connection(s)
I20240501 19:56:37.008374 14445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:56:37.009861 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 14445
I20240501 19:56:37.009997 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:56:37.010224 8865 external_mini_cluster.cc:1428] Setting key 44fdf6a381da945c4c6052f74e2a26a4
I20240501 19:56:37.011644 14513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.017871 14513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501: Bootstrap starting.
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:37.020371 14513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.021247 14513 log.cc:824] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:37.023294 14513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501: No bootstrap required, opened a new log
I20240501 19:56:37.023397 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:37.011453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53102 (local address 127.8.168.126:33287)
0501 19:56:37.011853 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:37.011865 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:37.011908 (+ 43us) server_negotiation.cc:407] Connection header received
0501 19:56:37.012751 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:37.012779 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:37.013139 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:37.013540 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:37.014336 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.015511 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:37.016212 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.016505 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:37.018656 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:37.018720 (+ 64us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:37.018735 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:37.018781 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:37.020841 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:37.021237 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:37.021243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:37.021248 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:37.021361 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:37.021571 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:37.021574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:37.021576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:37.021952 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:37.022140 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:37.022406 (+ 266us) server_negotiation.cc:299] Negotiation successful
0501 19:56:37.022664 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20240501 19:56:37.026499 14513 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1f5d0597de4dc6b069bb6245aef501" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33287 } }
I20240501 19:56:37.026684 14513 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.026731 14513 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1f5d0597de4dc6b069bb6245aef501, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.027119 14513 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1f5d0597de4dc6b069bb6245aef501" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33287 } }
I20240501 19:56:37.027235 14513 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:37.027278 14513 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:37.027356 14513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.028144 14513 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1f5d0597de4dc6b069bb6245aef501" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33287 } }
I20240501 19:56:37.028435 14513 leader_election.cc:302] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca1f5d0597de4dc6b069bb6245aef501; no voters:
I20240501 19:56:37.028677 14513 leader_election.cc:288] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:37.028820 14518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:37.029014 14518 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [term 1 LEADER]: Becoming Leader. State: Replica: ca1f5d0597de4dc6b069bb6245aef501, State: Running, Role: LEADER
I20240501 19:56:37.029300 14518 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1f5d0597de4dc6b069bb6245aef501" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33287 } }
I20240501 19:56:37.029726 14513 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:37.030820 14520 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca1f5d0597de4dc6b069bb6245aef501. Latest consensus state: current_term: 1 leader_uuid: "ca1f5d0597de4dc6b069bb6245aef501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1f5d0597de4dc6b069bb6245aef501" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33287 } } }
I20240501 19:56:37.031006 14520 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:37.031114 14519 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca1f5d0597de4dc6b069bb6245aef501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca1f5d0597de4dc6b069bb6245aef501" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33287 } } }
I20240501 19:56:37.031507 14519 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501 [sys.catalog]: This master's current role is: LEADER
W20240501 19:56:37.034152 14533 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:56:37.034221 14533 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:56:37.034304 14532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:37.034982 14532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:37.039937 14532 catalog_manager.cc:1347] Generated new cluster ID: bef15ac73d07482bbbacddb5b6effdaa
I20240501 19:56:37.040023 14532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:37.061230 14532 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:37.062214 14532 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:37.073349 14532 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ca1f5d0597de4dc6b069bb6245aef501: Generated new TSK 0
I20240501 19:56:37.074113 14532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
I20240501 19:56:37.102226 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33287
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:37.210763 14539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:37.211004 14539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:37.211066 14539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:37.214169 14539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:37.214238 14539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:37.214332 14539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:56:37.218544 14539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:37.219467 14539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:37.220324 14539 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:37.228410 14544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:37.229115 14545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:37.229729 14547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:37.230487 14539 server_base.cc:1034] running on GCE node
I20240501 19:56:37.231616 14539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:37.232863 14539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:37.234023 14539 hybrid_clock.cc:648] HybridClock initialized: now 1714593397234002 us; error 55 us; skew 500 ppm
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593397, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:37.236757 14539 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 19:56:37.237766 14539 webserver.cc:460] Webserver started at http://127.8.168.65:33841/ using document root <none> and password file <none>
I20240501 19:56:37.238315 14539 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:37.238373 14539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:37.238548 14539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:37.240403 14539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "1ca727ab88184e7abd0a13751ba38e6d"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-tzct"
server_key: "7a6f2df0a6491a6ccdbe79ed72c12296"
server_key_iv: "753e1eaa840f927a1848968fad79ff4c"
server_key_version: "encryptionkey@0"
I20240501 19:56:37.240911 14539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "1ca727ab88184e7abd0a13751ba38e6d"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-tzct"
server_key: "7a6f2df0a6491a6ccdbe79ed72c12296"
server_key_iv: "753e1eaa840f927a1848968fad79ff4c"
server_key_version: "encryptionkey@0"
I20240501 19:56:37.244333 14539 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 19:56:37.246604 14554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.247617 14539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:37.247735 14539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "1ca727ab88184e7abd0a13751ba38e6d"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-tzct"
server_key: "7a6f2df0a6491a6ccdbe79ed72c12296"
server_key_iv: "753e1eaa840f927a1848968fad79ff4c"
server_key_version: "encryptionkey@0"
I20240501 19:56:37.247826 14539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:37.272012 14539 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:37.275216 14539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:37.275385 14539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:37.275943 14539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:37.276809 14539 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:37.276860 14539 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.276922 14539 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:37.276947 14539 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.285667 14539 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:34521
I20240501 19:56:37.285697 14667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:34521 every 8 connection(s)
I20240501 19:56:37.286630 14539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 19:56:37.288347 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 14539
I20240501 19:56:37.288456 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 19:56:37.288762 8865 external_mini_cluster.cc:1428] Setting key 504507da8c633046e79453c758eb08bc
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593397, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.168.66@KRBTEST.COM; defaulting to no policy
I20240501 19:56:37.299002 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:37.288307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:36993 (local address 127.8.168.126:33287)
0501 19:56:37.288434 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:37.288438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:37.289558 (+ 1120us) server_negotiation.cc:407] Connection header received
0501 19:56:37.290317 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:37.290320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:37.290374 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:37.290453 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:37.292085 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.292589 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:37.293253 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.293437 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:37.295737 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:37.295756 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:37.295759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:37.295783 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:37.297166 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:37.297620 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:37.297624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:37.297626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:37.297672 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:37.297922 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:37.297926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:37.297928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:37.298108 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:37.298198 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:37.298687 (+ 489us) server_negotiation.cc:299] Negotiation successful
0501 19:56:37.298821 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.66@KRBTEST.COM" created.
I20240501 19:56:37.299654 14670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:37.288715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33287 (local address 127.8.168.65:36993)
0501 19:56:37.289216 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:37.289260 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:37.290102 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:37.290598 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:37.290606 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:37.291340 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:37.291935 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:37.291949 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.292749 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:37.292754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:37.293143 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:37.293150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.293367 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:37.293972 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:37.294001 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:37.295587 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:37.297271 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:37.297277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:37.297289 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:37.297528 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:37.297751 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:37.297754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:37.297756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:37.297837 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:37.298209 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:37.298216 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:37.298459 (+ 243us) client_negotiation.cc:769] Sending connection context
0501 19:56:37.298636 (+ 177us) client_negotiation.cc:240] Negotiation successful
0501 19:56:37.298910 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":124,"threads_started":1}
I20240501 19:56:37.300884 14668 heartbeater.cc:347] Connected to a master server at 127.8.168.126:33287
I20240501 19:56:37.301160 14668 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:37.301754 14668 heartbeater.cc:511] Master 127.8.168.126:33287 requested a full tablet report, sending...
I20240501 19:56:37.303314 14477 ts_manager.cc:194] Registered new tserver with Master: 1ca727ab88184e7abd0a13751ba38e6d (127.8.168.65:34521)
I20240501 19:56:37.304469 14477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:36993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
I20240501 19:56:37.313959 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33287
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:37.422547 14673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:37.422827 14673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:37.422915 14673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:37.426103 14673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:37.426179 14673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:37.426290 14673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:56:37.430650 14673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:37.431639 14673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:37.432518 14673 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:37.440447 14679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:37.441205 14673 server_base.cc:1034] running on GCE node
W20240501 19:56:37.441565 14681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:37.442098 14678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:37.442466 14673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:37.442991 14673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:37.444137 14673 hybrid_clock.cc:648] HybridClock initialized: now 1714593397444118 us; error 37 us; skew 500 ppm
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593397, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:37.446624 14673 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 19:56:37.447469 14673 webserver.cc:460] Webserver started at http://127.8.168.66:43805/ using document root <none> and password file <none>
I20240501 19:56:37.447983 14673 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:37.448064 14673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:37.448228 14673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:37.449949 14673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "d6c50f8049794bfa8f48af4e6d4dfbcd"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-tzct"
server_key: "44310c670df7322eae4d63e34fae2792"
server_key_iv: "3f82a5d36a2fa21d2d0cd44d33faa58f"
server_key_version: "encryptionkey@0"
I20240501 19:56:37.450393 14673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "d6c50f8049794bfa8f48af4e6d4dfbcd"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-tzct"
server_key: "44310c670df7322eae4d63e34fae2792"
server_key_iv: "3f82a5d36a2fa21d2d0cd44d33faa58f"
server_key_version: "encryptionkey@0"
I20240501 19:56:37.453631 14673 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:37.456333 14688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.457466 14673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:37.457569 14673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "d6c50f8049794bfa8f48af4e6d4dfbcd"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-tzct"
server_key: "44310c670df7322eae4d63e34fae2792"
server_key_iv: "3f82a5d36a2fa21d2d0cd44d33faa58f"
server_key_version: "encryptionkey@0"
I20240501 19:56:37.457664 14673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:37.477739 14673 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:37.480633 14673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:37.480818 14673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:37.481381 14673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:37.482165 14673 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:37.482213 14673 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.482250 14673 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:37.482280 14673 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.490482 14673 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:36739
I20240501 19:56:37.490506 14801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:36739 every 8 connection(s)
I20240501 19:56:37.491379 14673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 19:56:37.498951 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 14673
I20240501 19:56:37.499056 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593397, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:37.499281 8865 external_mini_cluster.cc:1428] Setting key 6e1b264d27dd1804846749c965840db8
I20240501 19:56:37.503085 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:37.493066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:44513 (local address 127.8.168.126:33287)
0501 19:56:37.493211 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:37.493216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:37.493923 (+ 707us) server_negotiation.cc:407] Connection header received
0501 19:56:37.494669 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:37.494672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:37.494721 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:37.494796 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:37.496341 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.496872 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:37.497440 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.497558 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:37.499740 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:37.499762 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:37.499765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:37.499791 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:37.501215 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:37.501684 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:37.501687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:37.501688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:37.501732 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:37.501985 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:37.501988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:37.501990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:37.502140 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:37.502223 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:37.502806 (+ 583us) server_negotiation.cc:299] Negotiation successful
0501 19:56:37.502941 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20240501 19:56:37.503816 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:37.493381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33287 (local address 127.8.168.66:44513)
0501 19:56:37.493784 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:37.493820 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:37.494477 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:37.494941 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:37.494948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:37.495313 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:37.496223 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:37.496236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.496964 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:37.496967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:37.497355 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:37.497362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.497538 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:37.498077 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:37.498100 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:37.499549 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:37.501331 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:37.501337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:37.501349 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:37.501586 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:37.501810 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:37.501814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:37.501816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:37.501903 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:37.502237 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:37.502244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:37.502491 (+ 247us) client_negotiation.cc:769] Sending connection context
0501 19:56:37.502700 (+ 209us) client_negotiation.cc:240] Negotiation successful
0501 19:56:37.502988 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":98,"threads_started":1}
I20240501 19:56:37.505034 14802 heartbeater.cc:347] Connected to a master server at 127.8.168.126:33287
I20240501 19:56:37.505259 14802 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:37.505775 14802 heartbeater.cc:511] Master 127.8.168.126:33287 requested a full tablet report, sending...
I20240501 19:56:37.506815 14477 ts_manager.cc:194] Registered new tserver with Master: d6c50f8049794bfa8f48af4e6d4dfbcd (127.8.168.66:36739)
I20240501 19:56:37.507378 14477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:44513
WARNING: no policy specified for kudu/127.8.168.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
I20240501 19:56:37.523887 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33287
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:37.631551 14807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:37.631807 14807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:37.631884 14807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:37.634967 14807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:37.635038 14807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:37.635140 14807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:56:37.639328 14807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:37.640233 14807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:37.641101 14807 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:37.648934 14812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:37.649552 14807 server_base.cc:1034] running on GCE node
W20240501 19:56:37.650017 14815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:37.650512 14813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:37.650895 14807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:37.651461 14807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:37.652621 14807 hybrid_clock.cc:648] HybridClock initialized: now 1714593397652612 us; error 44 us; skew 500 ppm
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593397, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:37.655189 14807 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 19:56:37.656121 14807 webserver.cc:460] Webserver started at http://127.8.168.67:34817/ using document root <none> and password file <none>
I20240501 19:56:37.656621 14807 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:37.656670 14807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:37.656858 14807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:37.658607 14807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "6bd5c4983c7a4e02b38466bfd1de678d"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-tzct"
server_key: "9470742fac117d3f24cd6d96e6ea28f1"
server_key_iv: "13554d3cc2a096c35735c5afce830095"
server_key_version: "encryptionkey@0"
I20240501 19:56:37.659024 14807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "6bd5c4983c7a4e02b38466bfd1de678d"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-tzct"
server_key: "9470742fac117d3f24cd6d96e6ea28f1"
server_key_iv: "13554d3cc2a096c35735c5afce830095"
server_key_version: "encryptionkey@0"
I20240501 19:56:37.662254 14807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:37.664333 14822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.665315 14807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:37.665412 14807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "6bd5c4983c7a4e02b38466bfd1de678d"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-tzct"
server_key: "9470742fac117d3f24cd6d96e6ea28f1"
server_key_iv: "13554d3cc2a096c35735c5afce830095"
server_key_version: "encryptionkey@0"
I20240501 19:56:37.665508 14807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:37.686579 14807 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:37.689096 14807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:37.689244 14807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:37.689754 14807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:37.690510 14807 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:37.690553 14807 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.690593 14807 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:37.690620 14807 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.698541 14807 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:40889
I20240501 19:56:37.698577 14935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:40889 every 8 connection(s)
I20240501 19:56:37.699448 14807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593397, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:37.709481 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 14807
I20240501 19:56:37.709633 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 19:56:37.709921 8865 external_mini_cluster.cc:1428] Setting key be5a5e05863b57150ee747bcccc002db
I20240501 19:56:37.710995 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:37.701083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:44529 (local address 127.8.168.126:33287)
0501 19:56:37.701223 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:37.701228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:37.701893 (+ 665us) server_negotiation.cc:407] Connection header received
0501 19:56:37.702675 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:37.702679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:37.702730 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:37.702812 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:37.704381 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.704982 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:37.705565 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.705695 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:37.707778 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:37.707798 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:37.707801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:37.707827 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:37.709004 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:37.709610 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:37.709613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:37.709615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:37.709662 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:37.709931 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:37.709934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:37.709936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:37.710100 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:37.710185 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:37.710718 (+ 533us) server_negotiation.cc:299] Negotiation successful
0501 19:56:37.710838 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20240501 19:56:37.711630 14938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:37.701362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33287 (local address 127.8.168.67:44529)
0501 19:56:37.701753 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:37.701787 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:37.702466 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:37.702956 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:37.702964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:37.703341 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:37.704236 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:37.704250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.705085 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:37.705087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:37.705475 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:37.705482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.705648 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:37.706180 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:37.706203 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:37.707629 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:37.709130 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:37.709134 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:37.709144 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:37.709508 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:37.709750 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:37.709753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:37.709755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:37.709841 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:37.710198 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:37.710204 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:37.710462 (+ 258us) client_negotiation.cc:769] Sending connection context
0501 19:56:37.710670 (+ 208us) client_negotiation.cc:240] Negotiation successful
0501 19:56:37.710934 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":95,"threads_started":1}
I20240501 19:56:37.712714 14936 heartbeater.cc:347] Connected to a master server at 127.8.168.126:33287
I20240501 19:56:37.712950 14936 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:37.713447 14936 heartbeater.cc:511] Master 127.8.168.126:33287 requested a full tablet report, sending...
I20240501 19:56:37.714454 14477 ts_manager.cc:194] Registered new tserver with Master: 6bd5c4983c7a4e02b38466bfd1de678d (127.8.168.67:40889)
I20240501 19:56:37.715057 14477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:44529
I20240501 19:56:37.724041 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/test-admin.keytab.
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:37.739135 8865 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20240501 19:56:37.741828 14943 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:37.740505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.8.168.65:34521)
0501 19:56:37.740870 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:37.740876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:37.740900 (+ 24us) server_negotiation.cc:407] Connection header received
0501 19:56:37.740955 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:37.740960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:37.741140 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:37.741260 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:37.741649 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41478: BlockingRecv error: recv got EOF from 127.0.0.1:41478 (error 108)
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":143,"threads_started":1}
W20240501 19:56:37.741966 8865 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/krb5cc' of type FILE
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:37.742949 8865 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:37.743793 8865 init.cc:295] Successfully reacquired a new kerberos TGT
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.65@KRBTEST.COM
I20240501 19:56:37.752315 14943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:37.744686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.8.168.65:34521)
0501 19:56:37.744816 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:37.744820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:37.744872 (+ 52us) server_negotiation.cc:407] Connection header received
0501 19:56:37.744992 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:37.744996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:37.745050 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:37.745116 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:37.745751 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.746485 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:37.747114 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.747352 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:37.749114 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:37.749138 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:37.749151 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:37.749177 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:37.750719 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:37.751117 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:37.751122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:37.751123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:37.751210 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:37.751415 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:37.751418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:37.751420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:37.751615 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:37.751827 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:37.751998 (+ 171us) server_negotiation.cc:299] Negotiation successful
0501 19:56:37.752175 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20240501 19:56:37.755997 14943 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:37.755110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41494 (local address 127.8.168.65:34521)
0501 19:56:37.755226 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:37.755230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:37.755302 (+ 72us) server_negotiation.cc:407] Connection header received
0501 19:56:37.755424 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:37.755426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:37.755466 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:37.755536 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:37.755893 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41494: BlockingRecv error: recv got EOF from 127.0.0.1:41494 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20240501 19:56:37.756121 8865 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/krb5cc' of type FILE
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:37.757071 8865 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:37.757964 8865 init.cc:295] Successfully reacquired a new kerberos TGT
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.65@KRBTEST.COM
I20240501 19:56:37.765261 14943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:37.758777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.8.168.65:34521)
0501 19:56:37.758880 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:37.758884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:37.758942 (+ 58us) server_negotiation.cc:407] Connection header received
0501 19:56:37.759050 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:37.759054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:37.759093 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:37.759155 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:37.759709 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.760179 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:37.760800 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.760981 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:37.762762 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:37.762778 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:37.762780 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:37.762801 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:37.763979 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:37.764317 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:37.764321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:37.764322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:37.764362 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:37.764555 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:37.764558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:37.764560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:37.764680 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:37.764813 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:37.765013 (+ 200us) server_negotiation.cc:299] Negotiation successful
0501 19:56:37.765109 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20240501 19:56:37.767717 14943 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:37.766932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41516 (local address 127.8.168.65:34521)
0501 19:56:37.767050 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:37.767054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:37.767066 (+ 12us) server_negotiation.cc:407] Connection header received
0501 19:56:37.767171 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:37.767174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:37.767209 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:37.767267 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:37.767617 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41516: BlockingRecv error: recv got EOF from 127.0.0.1:41516 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20240501 19:56:37.767968 8865 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1714593353892612-8865-0/krb5kdc/krb5cc)
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:37.769002 8865 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:37.769814 8865 init.cc:295] Successfully reacquired a new kerberos TGT
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.65@KRBTEST.COM
I20240501 19:56:37.776713 14943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:37.770540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41520 (local address 127.8.168.65:34521)
0501 19:56:37.770651 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:37.770653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:37.770735 (+ 82us) server_negotiation.cc:407] Connection header received
0501 19:56:37.770817 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:37.770819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:37.770850 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:37.770911 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:37.771423 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.771871 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:37.772415 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:37.772537 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:37.774307 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:37.774323 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:37.774325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:37.774346 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:37.775609 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:37.775934 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:37.775936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:37.775937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:37.775971 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:37.776112 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:37.776114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:37.776115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:37.776230 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:37.776328 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:37.776465 (+ 137us) server_negotiation.cc:299] Negotiation successful
0501 19:56:37.776561 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240501 19:56:37.777673 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 14539
I20240501 19:56:37.782904 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 14673
I20240501 19:56:37.787343 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 14807
I20240501 19:56:37.791680 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 14445
2024-05-01T19:56:37Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1960 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14956](info): set up 2 sockets
May 01 19:56:37 dist-test-slave-tzct krb5kdc[14956](info): commencing operation
krb5kdc: starting...
W20240501 19:56:38.685170 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.873s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 19:56:38 dist-test-slave-tzct krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T19:56:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.126.keytab.
Entry for principal oryx/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.126.keytab.
I20240501 19:56:38.796442 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:45289
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.126.keytab
--principal=oryx/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:45289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:38.900161 14969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:38.900426 14969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:38.900496 14969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:38.903609 14969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:38.903679 14969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:38.903709 14969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:38.903733 14969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:38.903761 14969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:38.907847 14969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:45289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.126.keytab
--principal=oryx/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:45289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:38.908788 14969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:38.909636 14969 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:38.916575 14975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:38.917106 14974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:38.917680 14977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:38.918150 14969 server_base.cc:1034] running on GCE node
I20240501 19:56:38.918665 14969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:38.919621 14969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:38.920775 14969 hybrid_clock.cc:648] HybridClock initialized: now 1714593398920760 us; error 29 us; skew 500 ppm
May 01 19:56:38 dist-test-slave-tzct krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593398, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:38.923491 14969 init.cc:377] Logged in from keytab as oryx/127.8.168.126@KRBTEST.COM (short username oryx)
I20240501 19:56:38.924441 14969 webserver.cc:460] Webserver started at http://127.8.168.126:36185/ using document root <none> and password file <none>
I20240501 19:56:38.925004 14969 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:38.925058 14969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:38.925246 14969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:38.927026 14969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "9fd6981a1f214470ab279f88136f1b14"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-tzct"
server_key: "c4f68cf8c61336f00bbe72c7773f56a0"
server_key_iv: "a954cace853f6e617ddda366935603e8"
server_key_version: "encryptionkey@0"
I20240501 19:56:38.927549 14969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "9fd6981a1f214470ab279f88136f1b14"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-tzct"
server_key: "c4f68cf8c61336f00bbe72c7773f56a0"
server_key_iv: "a954cace853f6e617ddda366935603e8"
server_key_version: "encryptionkey@0"
I20240501 19:56:38.930855 14969 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:38.933074 14984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:38.934069 14969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:38.934168 14969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "9fd6981a1f214470ab279f88136f1b14"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-tzct"
server_key: "c4f68cf8c61336f00bbe72c7773f56a0"
server_key_iv: "a954cace853f6e617ddda366935603e8"
server_key_version: "encryptionkey@0"
I20240501 19:56:38.934262 14969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:38.954614 14969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:38.957463 14969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:38.957628 14969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:38.964017 14969 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:45289
I20240501 19:56:38.964033 15036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:45289 every 8 connection(s)
I20240501 19:56:38.965324 14969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:56:38.967749 15037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:38.971935 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 14969
I20240501 19:56:38.972061 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:56:38.972321 8865 external_mini_cluster.cc:1428] Setting key eedca6d2ec391cda219458ed5d157c8a
I20240501 19:56:38.972816 15037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14: Bootstrap starting.
I20240501 19:56:38.975011 15037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:38.975654 15037 log.cc:824] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:38.977321 15037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14: No bootstrap required, opened a new log
May 01 19:56:38 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM
I20240501 19:56:38.980703 15037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } }
I20240501 19:56:38.980966 15037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:38.981029 15037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fd6981a1f214470ab279f88136f1b14, State: Initialized, Role: FOLLOWER
I20240501 19:56:38.981505 15037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } }
I20240501 19:56:38.981637 15037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:38.981699 15037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:38.981807 15037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:38.982707 15037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } }
I20240501 19:56:38.983042 15037 leader_election.cc:302] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fd6981a1f214470ab279f88136f1b14; no voters:
I20240501 19:56:38.983317 15037 leader_election.cc:288] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:38.983415 15042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:38.983572 15042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [term 1 LEADER]: Becoming Leader. State: Replica: 9fd6981a1f214470ab279f88136f1b14, State: Running, Role: LEADER
I20240501 19:56:38.983911 15042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } }
I20240501 19:56:38.984494 15037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:38.985129 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:38.973609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.8.168.126:45289)
0501 19:56:38.974018 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:38.974028 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:38.974070 (+ 42us) server_negotiation.cc:407] Connection header received
0501 19:56:38.974634 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:38.974650 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:38.974926 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:38.975254 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:38.976169 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:38.976988 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:38.977689 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:38.977947 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:38.980311 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:38.980332 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:38.980343 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:38.980379 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:38.982273 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:38.982724 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:38.982728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:38.982732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:38.982808 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:38.983026 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:38.983029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:38.983041 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:38.983443 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:38.983668 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:38.984071 (+ 403us) server_negotiation.cc:299] Negotiation successful
0501 19:56:38.984355 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":137,"threads_started":1}
I20240501 19:56:38.985481 15044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fd6981a1f214470ab279f88136f1b14. Latest consensus state: current_term: 1 leader_uuid: "9fd6981a1f214470ab279f88136f1b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } }
I20240501 19:56:38.985592 15044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:38.985838 15043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fd6981a1f214470ab279f88136f1b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } }
I20240501 19:56:38.985945 15043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: This master's current role is: LEADER
W20240501 19:56:38.988876 15057 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:56:38.988948 15057 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:56:38.989017 15051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:38.989727 15051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:38.994755 15051 catalog_manager.cc:1347] Generated new cluster ID: cb2d94b3f8bd46e9bbe19926806e1c00
I20240501 19:56:38.994834 15051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:39.012074 15051 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:39.013084 15051 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:39.022384 15051 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14: Generated new TSK 0
I20240501 19:56:39.023241 15051 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.65.keytab.
Entry for principal oryx/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.65.keytab.
I20240501 19:56:39.056012 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:45289
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.65.keytab
--principal=oryx/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:39.163324 15063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:39.163571 15063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:39.163630 15063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:39.166664 15063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:39.166730 15063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:39.166805 15063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:56:39.170878 15063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.65.keytab
--principal=oryx/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:45289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:39.171739 15063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:39.172525 15063 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:39.180586 15068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:39.180688 15069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:39.181252 15071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:39.181777 15063 server_base.cc:1034] running on GCE node
I20240501 19:56:39.182219 15063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:39.182849 15063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:39.184000 15063 hybrid_clock.cc:648] HybridClock initialized: now 1714593399183985 us; error 32 us; skew 500 ppm
May 01 19:56:39 dist-test-slave-tzct krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593399, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:39.186883 15063 init.cc:377] Logged in from keytab as oryx/127.8.168.65@KRBTEST.COM (short username oryx)
I20240501 19:56:39.187821 15063 webserver.cc:460] Webserver started at http://127.8.168.65:38405/ using document root <none> and password file <none>
I20240501 19:56:39.188347 15063 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:39.188405 15063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:39.188603 15063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:39.190436 15063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "3247c2f74cd44a12a587f056979b798f"
format_stamp: "Formatted at 2024-05-01 19:56:39 on dist-test-slave-tzct"
server_key: "6e7d07928a6f4fa7cd182ab7565c5179"
server_key_iv: "55d8889cf3a869e43e5bc6a65bf1048a"
server_key_version: "encryptionkey@0"
I20240501 19:56:39.190909 15063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "3247c2f74cd44a12a587f056979b798f"
format_stamp: "Formatted at 2024-05-01 19:56:39 on dist-test-slave-tzct"
server_key: "6e7d07928a6f4fa7cd182ab7565c5179"
server_key_iv: "55d8889cf3a869e43e5bc6a65bf1048a"
server_key_version: "encryptionkey@0"
I20240501 19:56:39.194777 15063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 19:56:39.197292 15078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:39.198375 15063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:39.198488 15063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "3247c2f74cd44a12a587f056979b798f"
format_stamp: "Formatted at 2024-05-01 19:56:39 on dist-test-slave-tzct"
server_key: "6e7d07928a6f4fa7cd182ab7565c5179"
server_key_iv: "55d8889cf3a869e43e5bc6a65bf1048a"
server_key_version: "encryptionkey@0"
I20240501 19:56:39.198585 15063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:39.217104 15063 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:39.219806 15063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:39.220034 15063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:39.220590 15063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:39.221460 15063 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:39.221509 15063 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:39.221563 15063 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:39.221588 15063 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:39.230134 15063 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:46591
I20240501 19:56:39.230173 15191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:46591 every 8 connection(s)
I20240501 19:56:39.231062 15063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 19:56:39.231720 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 15063
I20240501 19:56:39.231853 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 19:56:39.232096 8865 external_mini_cluster.cc:1428] Setting key 44572db8a045658de732009d7c767b53
May 01 19:56:39 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593399, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.8.168.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.168.66@KRBTEST.COM" created.
I20240501 19:56:39.245527 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.233059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:58313 (local address 127.8.168.126:45289)
0501 19:56:39.233296 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:39.233301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:39.234330 (+ 1029us) server_negotiation.cc:407] Connection header received
0501 19:56:39.235635 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:39.235638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:39.235691 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:39.235779 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:39.237230 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.237883 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.238691 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.238950 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.241512 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:39.241536 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:39.241539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:39.241568 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:39.243216 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.243833 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.243836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.243837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.243932 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.244225 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.244229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.244231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.244450 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:39.244564 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:39.245263 (+ 699us) server_negotiation.cc:299] Negotiation successful
0501 19:56:39.245383 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20240501 19:56:39.246294 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.233516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.65:58313)
0501 19:56:39.234180 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:39.234224 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:39.235362 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:39.235973 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:39.235983 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:39.236367 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:39.237069 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.237088 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.238052 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.238056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:39.238574 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.238584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.238824 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.239583 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:39.239612 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:39.241301 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:39.243367 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.243372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.243382 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.243714 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.244016 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.244019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.244020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.244122 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.244562 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:39.244569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:39.244939 (+ 370us) client_negotiation.cc:769] Sending connection context
0501 19:56:39.245185 (+ 246us) client_negotiation.cc:240] Negotiation successful
0501 19:56:39.245486 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":106,"threads_started":1}
I20240501 19:56:39.247555 15192 heartbeater.cc:347] Connected to a master server at 127.8.168.126:45289
I20240501 19:56:39.247814 15192 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:39.248490 15192 heartbeater.cc:511] Master 127.8.168.126:45289 requested a full tablet report, sending...
I20240501 19:56:39.250097 15001 ts_manager.cc:194] Registered new tserver with Master: 3247c2f74cd44a12a587f056979b798f (127.8.168.65:46591)
I20240501 19:56:39.251258 15001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:58313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.66.keytab.
Entry for principal oryx/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.66.keytab.
I20240501 19:56:39.257302 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:45289
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.66.keytab
--principal=oryx/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:39.362713 15197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:39.362987 15197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:39.363063 15197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:39.366145 15197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:39.366214 15197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:39.366312 15197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:56:39.370296 15197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.66.keytab
--principal=oryx/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:45289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:39.371178 15197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:39.372012 15197 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:39.379979 15203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:39.380414 15205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:39.380996 15202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:39.381441 15197 server_base.cc:1034] running on GCE node
I20240501 19:56:39.381836 15197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:39.382426 15197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:39.383563 15197 hybrid_clock.cc:648] HybridClock initialized: now 1714593399383545 us; error 37 us; skew 500 ppm
May 01 19:56:39 dist-test-slave-tzct krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593399, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:39.386258 15197 init.cc:377] Logged in from keytab as oryx/127.8.168.66@KRBTEST.COM (short username oryx)
I20240501 19:56:39.387176 15197 webserver.cc:460] Webserver started at http://127.8.168.66:36205/ using document root <none> and password file <none>
I20240501 19:56:39.387688 15197 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:39.387738 15197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:39.387895 15197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:39.389674 15197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "2e74f065d34341d8a34c0b20761a558f"
format_stamp: "Formatted at 2024-05-01 19:56:39 on dist-test-slave-tzct"
server_key: "99764d94e2c5e51ff779af19e4a9c520"
server_key_iv: "1df0af08a762b99f081dffa88fc7d473"
server_key_version: "encryptionkey@0"
I20240501 19:56:39.390097 15197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "2e74f065d34341d8a34c0b20761a558f"
format_stamp: "Formatted at 2024-05-01 19:56:39 on dist-test-slave-tzct"
server_key: "99764d94e2c5e51ff779af19e4a9c520"
server_key_iv: "1df0af08a762b99f081dffa88fc7d473"
server_key_version: "encryptionkey@0"
I20240501 19:56:39.393302 15197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:39.395442 15212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:39.396447 15197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:39.396554 15197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "2e74f065d34341d8a34c0b20761a558f"
format_stamp: "Formatted at 2024-05-01 19:56:39 on dist-test-slave-tzct"
server_key: "99764d94e2c5e51ff779af19e4a9c520"
server_key_iv: "1df0af08a762b99f081dffa88fc7d473"
server_key_version: "encryptionkey@0"
I20240501 19:56:39.396649 15197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:39.413172 15197 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:39.415853 15197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:39.416018 15197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:39.416579 15197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:39.417374 15197 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:39.417420 15197 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:39.417483 15197 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:39.417526 15197 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:39.425676 15197 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:41839
I20240501 19:56:39.425706 15325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:41839 every 8 connection(s)
I20240501 19:56:39.426616 15197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 19:56:39.433013 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 15197
I20240501 19:56:39.433147 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 19:56:39.433430 8865 external_mini_cluster.cc:1428] Setting key b35c67bec8efcf35dd538533ce83ef0a
May 01 19:56:39 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593399, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM
I20240501 19:56:39.438745 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.428252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:33735 (local address 127.8.168.126:45289)
0501 19:56:39.428415 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:39.428419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:39.429086 (+ 667us) server_negotiation.cc:407] Connection header received
0501 19:56:39.430275 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:39.430279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:39.430328 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:39.430400 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:39.431633 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.432183 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.432835 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.433019 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.435447 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:39.435463 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:39.435465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:39.435487 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:39.436817 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.437278 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.437281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.437283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.437334 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.437601 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.437604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.437606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.437768 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:39.437853 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:39.438505 (+ 652us) server_negotiation.cc:299] Negotiation successful
0501 19:56:39.438617 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20240501 19:56:39.439599 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.428545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.66:33735)
0501 19:56:39.428964 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:39.429001 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:39.430069 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:39.430554 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:39.430562 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:39.430932 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:39.431483 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.431496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.432278 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.432281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:39.432683 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.432710 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.432917 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.433564 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:39.433590 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:39.435279 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:39.436924 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.436929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.436938 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.437182 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.437424 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.437427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.437428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.437513 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.437910 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:39.437916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:39.438249 (+ 333us) client_negotiation.cc:769] Sending connection context
0501 19:56:39.438469 (+ 220us) client_negotiation.cc:240] Negotiation successful
0501 19:56:39.438761 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":125,"threads_started":1}
I20240501 19:56:39.440667 15326 heartbeater.cc:347] Connected to a master server at 127.8.168.126:45289
I20240501 19:56:39.440901 15326 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:39.441401 15326 heartbeater.cc:511] Master 127.8.168.126:45289 requested a full tablet report, sending...
I20240501 19:56:39.442384 15001 ts_manager.cc:194] Registered new tserver with Master: 2e74f065d34341d8a34c0b20761a558f (127.8.168.66:41839)
WARNING: no policy specified for oryx/127.8.168.67@KRBTEST.COM; defaulting to no policy
I20240501 19:56:39.443022 15001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:33735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.168.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.67.keytab.
Entry for principal oryx/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.67.keytab.
I20240501 19:56:39.457279 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:45289
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.67.keytab
--principal=oryx/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:39.563208 15331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:39.563460 15331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:39.563529 15331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:39.566629 15331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:39.566692 15331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:39.566787 15331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:56:39.570729 15331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.67.keytab
--principal=oryx/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:45289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:39.571563 15331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:39.572369 15331 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:39.580052 15336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:39.580557 15337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:39.581104 15339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:39.581619 15331 server_base.cc:1034] running on GCE node
I20240501 19:56:39.581982 15331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:39.582501 15331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:39.583647 15331 hybrid_clock.cc:648] HybridClock initialized: now 1714593399583640 us; error 28 us; skew 500 ppm
May 01 19:56:39 dist-test-slave-tzct krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593399, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:39.586134 15331 init.cc:377] Logged in from keytab as oryx/127.8.168.67@KRBTEST.COM (short username oryx)
I20240501 19:56:39.586994 15331 webserver.cc:460] Webserver started at http://127.8.168.67:42119/ using document root <none> and password file <none>
I20240501 19:56:39.587494 15331 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:39.587549 15331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:39.587710 15331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:39.589432 15331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "2522fdb3bfeb457e9a395935914b896c"
format_stamp: "Formatted at 2024-05-01 19:56:39 on dist-test-slave-tzct"
server_key: "b22d74993a5ff016a1c9f2b579dadf41"
server_key_iv: "1480aaa14336be89f6071f876a54dbaa"
server_key_version: "encryptionkey@0"
I20240501 19:56:39.589886 15331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "2522fdb3bfeb457e9a395935914b896c"
format_stamp: "Formatted at 2024-05-01 19:56:39 on dist-test-slave-tzct"
server_key: "b22d74993a5ff016a1c9f2b579dadf41"
server_key_iv: "1480aaa14336be89f6071f876a54dbaa"
server_key_version: "encryptionkey@0"
I20240501 19:56:39.592995 15331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 19:56:39.595031 15346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:39.595996 15331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:39.596096 15331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "2522fdb3bfeb457e9a395935914b896c"
format_stamp: "Formatted at 2024-05-01 19:56:39 on dist-test-slave-tzct"
server_key: "b22d74993a5ff016a1c9f2b579dadf41"
server_key_iv: "1480aaa14336be89f6071f876a54dbaa"
server_key_version: "encryptionkey@0"
I20240501 19:56:39.596180 15331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:39.606184 15331 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:39.609083 15331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:39.609241 15331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:39.609758 15331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:39.610515 15331 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:39.610563 15331 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:39.610611 15331 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:39.610638 15331 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:39.618642 15331 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:37273
I20240501 19:56:39.618669 15459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:37273 every 8 connection(s)
I20240501 19:56:39.619511 15331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 19:56:39.622246 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 15331
I20240501 19:56:39.622342 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 19:56:39.622555 8865 external_mini_cluster.cc:1428] Setting key 98075eb31075da3c8be3d89f53f0f56b
May 01 19:56:39 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593399, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM
I20240501 19:56:39.630810 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.621074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:51139 (local address 127.8.168.126:45289)
0501 19:56:39.621216 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:39.621220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:39.621830 (+ 610us) server_negotiation.cc:407] Connection header received
0501 19:56:39.623053 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:39.623056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:39.623112 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:39.623187 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:39.624417 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.624952 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.625575 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.625750 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.627758 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:39.627774 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:39.627776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:39.627799 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:39.629104 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.629542 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.629545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.629547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.629588 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.629828 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.629831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.629833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.629981 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:39.630064 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:39.630527 (+ 463us) server_negotiation.cc:299] Negotiation successful
0501 19:56:39.630646 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20240501 19:56:39.631387 15462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.621317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.67:51139)
0501 19:56:39.621701 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:39.621735 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:39.622852 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:39.623354 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:39.623362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:39.623735 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:39.624284 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.624296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.625084 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.625087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:39.625463 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.625470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.625674 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.626257 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:39.626282 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:39.627617 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:39.629207 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.629213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.629224 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.629445 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.629664 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.629667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.629669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.629748 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.630075 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:39.630080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:39.630303 (+ 223us) client_negotiation.cc:769] Sending connection context
0501 19:56:39.630483 (+ 180us) client_negotiation.cc:240] Negotiation successful
0501 19:56:39.630745 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":101,"threads_started":1}
I20240501 19:56:39.632365 15460 heartbeater.cc:347] Connected to a master server at 127.8.168.126:45289
I20240501 19:56:39.632582 15460 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:39.633101 15460 heartbeater.cc:511] Master 127.8.168.126:45289 requested a full tablet report, sending...
I20240501 19:56:39.634029 15001 ts_manager.cc:194] Registered new tserver with Master: 2522fdb3bfeb457e9a395935914b896c (127.8.168.67:37273)
I20240501 19:56:39.634619 15001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:51139
I20240501 19:56:39.635902 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.168.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.168.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.168.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.125.keytab.
Entry for principal oryx/127.8.168.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.125.keytab.
I20240501 19:56:39.659610 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.125:36409
--webserver_interface=127.8.168.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.125.keytab
--principal=oryx/127.8.168.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:45289,127.8.168.125:36409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:39.762144 15465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:39.762363 15465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:39.762419 15465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:39.765360 15465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:39.765414 15465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:39.765434 15465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:39.765451 15465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:39.765484 15465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:39.769361 15465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:45289,127.8.168.125:36409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.125.keytab
--principal=oryx/127.8.168.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.125:36409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.168.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
W20240501 19:56:39.769558 15465 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.168.126:45289,127.8.168.125:36409'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20240501 19:56:39.770246 15465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:39.771060 15465 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:39.777770 15473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:39.778192 15470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:39.778746 15471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:39.779284 15465 server_base.cc:1034] running on GCE node
I20240501 19:56:39.779754 15465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:39.780606 15465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:39.781746 15465 hybrid_clock.cc:648] HybridClock initialized: now 1714593399781747 us; error 39 us; skew 500 ppm
May 01 19:56:39 dist-test-slave-tzct krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593399, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:39.784171 15465 init.cc:377] Logged in from keytab as oryx/127.8.168.125@KRBTEST.COM (short username oryx)
I20240501 19:56:39.785082 15465 webserver.cc:460] Webserver started at http://127.8.168.125:33439/ using document root <none> and password file <none>
I20240501 19:56:39.785605 15465 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:39.785679 15465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:39.785871 15465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:39.787583 15465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/data/instance:
uuid: "a88d3cb5c6774a43acf84bda9c2362e1"
format_stamp: "Formatted at 2024-05-01 19:56:39 on dist-test-slave-tzct"
server_key: "017038339f375249ba4b25c6e4c47338"
server_key_iv: "24b7085893682540a1bd4d5c0643b9c3"
server_key_version: "encryptionkey@0"
I20240501 19:56:39.788018 15465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/wal/instance:
uuid: "a88d3cb5c6774a43acf84bda9c2362e1"
format_stamp: "Formatted at 2024-05-01 19:56:39 on dist-test-slave-tzct"
server_key: "017038339f375249ba4b25c6e4c47338"
server_key_iv: "24b7085893682540a1bd4d5c0643b9c3"
server_key_version: "encryptionkey@0"
I20240501 19:56:39.791221 15465 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 19:56:39.793378 15480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:39.794334 15465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:39.794430 15465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/wal
uuid: "a88d3cb5c6774a43acf84bda9c2362e1"
format_stamp: "Formatted at 2024-05-01 19:56:39 on dist-test-slave-tzct"
server_key: "017038339f375249ba4b25c6e4c47338"
server_key_iv: "24b7085893682540a1bd4d5c0643b9c3"
server_key_version: "encryptionkey@0"
I20240501 19:56:39.794515 15465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:39.815747 15465 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:39.818462 15465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:39.818620 15465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:39.824905 15465 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.125:36409
I20240501 19:56:39.824923 15532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.125:36409 every 8 connection(s)
I20240501 19:56:39.826082 15465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/data/info.pb
I20240501 19:56:39.828459 15533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.831596 15533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:56:39.834575 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 15465
May 01 19:56:39 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593399, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.125@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM
May 01 19:56:39 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.168.125@KRBTEST.COM
I20240501 19:56:39.843124 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.833238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.8.168.126:45289)
0501 19:56:39.833360 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:39.833363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:39.834041 (+ 678us) server_negotiation.cc:407] Connection header received
0501 19:56:39.834842 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:39.834846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:39.834906 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:39.834990 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:39.836306 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.836849 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.837450 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.837628 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.839793 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:39.839811 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:39.839813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:39.839838 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:39.841257 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.841710 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.841714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.841716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.841763 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.841990 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.841993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.841995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.842206 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:39.842286 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:39.842836 (+ 550us) server_negotiation.cc:299] Negotiation successful
0501 19:56:39.842957 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240501 19:56:39.843705 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.835312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43124 (local address 127.8.168.125:36409)
0501 19:56:39.835610 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:39.835615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:39.835638 (+ 23us) server_negotiation.cc:407] Connection header received
0501 19:56:39.835771 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:39.835776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:39.835913 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:39.836040 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:39.836822 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.837528 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.838149 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.838355 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.840473 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:39.840491 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:39.840501 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:39.840520 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:39.842025 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.842357 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.842361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.842362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.842405 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.842623 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.842624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.842626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.842790 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:39.842974 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:39.843200 (+ 226us) server_negotiation.cc:299] Negotiation successful
0501 19:56:39.843332 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":64,"threads_started":1}
I20240501 19:56:39.843708 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.833461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.0.0.1:33396)
0501 19:56:39.833873 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:39.833909 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:39.834610 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:39.835150 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:39.835161 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:39.835619 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:39.836167 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.836180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.836948 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.836951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:39.837344 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.837354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.837544 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.838143 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:39.838171 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:39.839656 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:39.841354 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.841360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.841366 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.841618 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.841842 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.841844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.841845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.841912 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.842311 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:39.842317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:39.842591 (+ 274us) client_negotiation.cc:769] Sending connection context
0501 19:56:39.842771 (+ 180us) client_negotiation.cc:240] Negotiation successful
0501 19:56:39.843002 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":105,"threads_started":1}
I20240501 19:56:39.845345 15533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } has no permanent_uuid. Determining permanent_uuid...
May 01 19:56:39 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593399, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.125@KRBTEST.COM for oryx/127.8.168.125@KRBTEST.COM
I20240501 19:56:39.851260 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.845813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43132 (local address 127.8.168.125:36409)
0501 19:56:39.845914 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:39.845917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:39.845927 (+ 10us) server_negotiation.cc:407] Connection header received
0501 19:56:39.846029 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:39.846031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:39.846071 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:39.846130 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:39.846487 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.846909 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.847397 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.847518 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.849031 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:39.849052 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:39.849054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:39.849072 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:39.850097 (+ 1025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.850400 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.850403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.850405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.850437 (+ 32us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.850630 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.850633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.850635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.850750 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:39.850829 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:39.851044 (+ 215us) server_negotiation.cc:299] Negotiation successful
0501 19:56:39.851144 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":37}
I20240501 19:56:39.851246 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.845767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36409 (local address 127.0.0.1:43132)
0501 19:56:39.845873 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:39.845886 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:39.845967 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:39.846149 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:39.846151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:39.846261 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:39.846412 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.846417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.846986 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.846988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:39.847325 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.847331 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.847409 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.847797 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:39.847808 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:39.848931 (+ 1123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:39.850187 (+ 1256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.850190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.850192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.850326 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.850505 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.850508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.850511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.850566 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.850827 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:39.850830 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:39.850897 (+ 67us) client_negotiation.cc:769] Sending connection context
0501 19:56:39.850977 (+ 80us) client_negotiation.cc:240] Negotiation successful
0501 19:56:39.851087 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20240501 19:56:39.854225 15533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: Bootstrap starting.
I20240501 19:56:39.856215 15533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:39.856910 15533 log.cc:824] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:39.858500 15533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: No bootstrap required, opened a new log
I20240501 19:56:39.860970 15533 raft_consensus.cc:348] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } }
I20240501 19:56:39.861128 15533 raft_consensus.cc:374] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:39.861186 15533 raft_consensus.cc:729] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88d3cb5c6774a43acf84bda9c2362e1, State: Initialized, Role: FOLLOWER
I20240501 19:56:39.861590 15533 consensus_queue.cc:260] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } }
I20240501 19:56:39.861994 15539 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } } }
I20240501 19:56:39.862107 15539 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:56:39.862602 15533 sys_catalog.cc:564] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: configured and running, proceeding with master startup.
W20240501 19:56:39.866767 15550 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:56:39.866849 15550 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:56:39.868324 15465 master_runner.cc:428] Detected that this master a88d3cb5c6774a43acf84bda9c2362e1 is joining an existing cluster
I20240501 19:56:39.868372 15465 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.168.125:36409
I20240501 19:56:39.878028 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.870660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.8.168.126:45289)
0501 19:56:39.870785 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:39.870789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:39.870884 (+ 95us) server_negotiation.cc:407] Connection header received
0501 19:56:39.871003 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:39.871005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:39.871046 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:39.871107 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:39.871846 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.872350 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.873227 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.873372 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.874421 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:39.874447 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:39.874451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:39.874476 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:39.875990 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.876534 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.876538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.876539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.876588 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.876884 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.876887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.876889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.877062 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:39.877154 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:39.877748 (+ 594us) server_negotiation.cc:299] Negotiation successful
0501 19:56:39.877859 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240501 19:56:39.878218 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.870387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.0.0.1:33400)
0501 19:56:39.870799 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:39.870812 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:39.870917 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:39.871141 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:39.871144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:39.871379 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:39.871704 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.871713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.872452 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.872456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:39.873116 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.873127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.873239 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.873837 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:39.873852 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:39.874295 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:39.876123 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.876126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.876129 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.876432 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.876683 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.876686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.876688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.876803 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.877440 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:39.877445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:39.877532 (+ 87us) client_negotiation.cc:769] Sending connection context
0501 19:56:39.877669 (+ 137us) client_negotiation.cc:240] Negotiation successful
0501 19:56:39.877816 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":153,"threads_started":1}
I20240501 19:56:39.879540 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.870489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.8.168.125:36409)
0501 19:56:39.870604 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:39.870607 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:39.873675 (+ 3068us) server_negotiation.cc:407] Connection header received
0501 19:56:39.873844 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:39.873847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:39.873882 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:39.873942 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:39.874437 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.875093 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.875712 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.876151 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.876652 (+ 501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:39.876668 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:39.876671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:39.876696 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:39.878222 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.878597 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.878600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.878602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.878638 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.878831 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.878833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.878835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.878974 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:39.879034 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:39.879292 (+ 258us) server_negotiation.cc:299] Negotiation successful
0501 19:56:39.879406 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":29}
I20240501 19:56:39.879552 15556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.870387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36409 (local address 127.0.0.1:43136)
0501 19:56:39.873607 (+ 3220us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:39.873620 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:39.873727 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:39.873970 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:39.873973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:39.874161 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:39.874348 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.874355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.875201 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.875205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:39.875610 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.875619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.875717 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.876172 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:39.876188 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:39.876546 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:39.878325 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.878327 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.878329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.878513 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.878718 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.878720 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.878721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.878762 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.879084 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:39.879086 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:39.879146 (+ 60us) client_negotiation.cc:769] Sending connection context
0501 19:56:39.879235 (+ 89us) client_negotiation.cc:240] Negotiation successful
0501 19:56:39.879353 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3151,"mutex_wait_us":35,"spinlock_wait_cycles":38912,"thread_start_us":138,"threads_started":1}
May 01 19:56:39 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593398, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.126@KRBTEST.COM for oryx/127.8.168.125@KRBTEST.COM
WARNING: no policy specified for oryx/127.8.168.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.168.124@KRBTEST.COM" created.
I20240501 19:56:39.908124 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.882566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43144 (local address 127.8.168.125:36409)
0501 19:56:39.882674 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:39.882678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:39.882902 (+ 224us) server_negotiation.cc:407] Connection header received
0501 19:56:39.883095 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:39.883098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:39.883147 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:39.883226 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:39.883903 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.884584 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.885229 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.885410 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.887291 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:39.887308 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:39.887311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:39.887335 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:39.906269 (+ 18934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.906936 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.906940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.906941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.906989 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.907213 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.907216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.907218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.907374 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:39.907460 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:39.907808 (+ 348us) server_negotiation.cc:299] Negotiation successful
0501 19:56:39.907920 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240501 19:56:39.908200 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.882492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36409 (local address 127.0.0.1:43144)
0501 19:56:39.882791 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:39.882830 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:39.882956 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:39.883255 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:39.883259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:39.883604 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:39.883800 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.883807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.884679 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.884681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:39.885138 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.885146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.885229 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.885670 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:39.885682 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:39.887176 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:39.906480 (+ 19304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.906496 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.906502 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.906807 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.907075 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.907077 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.907079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.907130 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.907485 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:39.907491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:39.907607 (+ 116us) client_negotiation.cc:769] Sending connection context
0501 19:56:39.907741 (+ 134us) client_negotiation.cc:240] Negotiation successful
0501 19:56:39.908021 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":74,"threads_started":1}
I20240501 19:56:39.909787 15001 catalog_manager.cc:6968] Initiating ChangeConfig request to add master 127.8.168.125:36409: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1"
member_type: NON_VOTER
last_known_addr {
host: "127.8.168.125"
port: 36409
}
attrs {
promote: true
}
}
dest_uuid: "9fd6981a1f214470ab279f88136f1b14"
cas_config_opid_index: -1
I20240501 19:56:39.910341 15001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: NON_VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: true } }
I20240501 19:56:39.911782 15561 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a88d3cb5c6774a43acf84bda9c2362e1 (127.8.168.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: NON_VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: true } } }
I20240501 19:56:39.912634 15507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:39.913655 15561 catalog_manager.cc:6972] Successfully completed master ChangeConfig request to add master 127.8.168.125:36409
I20240501 19:56:39.914076 15465 master.cc:537] Master@127.8.168.125:36409 shutting down...
I20240501 19:56:39.914371 15507 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 1 FOLLOWER]: Refusing update from remote peer 9fd6981a1f214470ab279f88136f1b14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:39.914500 15561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9fd6981a1f214470ab279f88136f1b14" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: NON_VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: true } } }
I20240501 19:56:39.914582 15561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:39.916414 15562 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: NON_VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:39.917979 15465 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:39.918228 15465 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:39.918329 15465 tablet_replica.cc:330] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: stopping tablet replica
W20240501 19:56:39.918931 14986 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20240501 19:56:39.919808 14986 consensus_peers.cc:485] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 -> Peer a88d3cb5c6774a43acf84bda9c2362e1 (127.8.168.125:36409): Couldn't send request to peer a88d3cb5c6774a43acf84bda9c2362e1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:39.922031 15465 master.cc:552] Master@127.8.168.125:36409 shutdown complete.
I20240501 19:56:39.922071 15465 master_runner.cc:305] Clearing existing system tablet
I20240501 19:56:39.922258 15465 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:39.922303 15465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.168.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.124.keytab.
Entry for principal oryx/127.8.168.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.124.keytab.
I20240501 19:56:39.923388 15465 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:39.923596 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.124:40517
--webserver_interface=127.8.168.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.124.keytab
--principal=oryx/127.8.168.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:45289,127.8.168.125:36409,127.8.168.124:40517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 19:56:39.924099 15567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:39.924264 15465 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:39.924384 15465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:39.924448 15465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/wal
uuid: "a88d3cb5c6774a43acf84bda9c2362e1"
format_stamp: "Formatted at 2024-05-01 19:56:39 on dist-test-slave-tzct"
server_key: "017038339f375249ba4b25c6e4c47338"
server_key_iv: "24b7085893682540a1bd4d5c0643b9c3"
server_key_version: "encryptionkey@0"
I20240501 19:56:39.926272 15465 ts_tablet_manager.cc:1875] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:39.927943 15465 ts_tablet_manager.cc:1888] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20240501 19:56:39.928014 15465 log.cc:1196] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20240501 19:56:39.928243 15465 ts_tablet_manager.cc:1909] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: Deleting consensus metadata
I20240501 19:56:39.928380 15465 master_runner.cc:315] Copying system tablet from 127.8.168.126:45289
I20240501 19:56:39.929587 15465 tablet_copy_client.cc:308] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: tablet copy: Beginning tablet copy session from remote peer at address 127.8.168.126:45289
I20240501 19:56:39.935990 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.930171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.8.168.126:45289)
0501 19:56:39.930295 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:39.930300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:39.930465 (+ 165us) server_negotiation.cc:407] Connection header received
0501 19:56:39.930602 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:39.930605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:39.930652 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:39.930717 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:39.931240 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.931733 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.932327 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.932511 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.933416 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:39.933440 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:39.933442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:39.933465 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:39.934704 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.935056 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.935059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.935060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.935100 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:39.935294 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:39.935297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:39.935299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:39.935443 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:39.935527 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:39.935748 (+ 221us) server_negotiation.cc:299] Negotiation successful
0501 19:56:39.935842 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240501 19:56:39.936007 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:39.930095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.0.0.1:33402)
0501 19:56:39.930383 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:39.930395 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:39.930507 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:39.930743 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:39.930746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:39.930936 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:39.931123 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.931129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.931832 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:39.931836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:39.932224 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:39.932232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:39.932331 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:39.933010 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:39.933022 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:39.933297 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:39.934803 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.934807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.934809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.934971 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.935173 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:39.935177 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:39.935179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:39.935220 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:39.935525 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:39.935528 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:39.935603 (+ 75us) client_negotiation.cc:769] Sending connection context
0501 19:56:39.935687 (+ 84us) client_negotiation.cc:240] Negotiation successful
0501 19:56:39.935815 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":89,"threads_started":1}
I20240501 19:56:39.936671 15021 tablet_copy_service.cc:140] P 9fd6981a1f214470ab279f88136f1b14: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a88d3cb5c6774a43acf84bda9c2362e1 ({username='oryx', principal='oryx/127.8.168.125@KRBTEST.COM'} at 127.0.0.1:33402)
I20240501 19:56:39.936848 15021 tablet_copy_service.cc:161] P 9fd6981a1f214470ab279f88136f1b14: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a88d3cb5c6774a43acf84bda9c2362e1 at {username='oryx', principal='oryx/127.8.168.125@KRBTEST.COM'} at 127.0.0.1:33402: session id = a88d3cb5c6774a43acf84bda9c2362e1-00000000000000000000000000000000
I20240501 19:56:39.938190 15021 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:39.939548 15465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20240501 19:56:39.942162 15465 tablet_copy_client.cc:783] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:39.942416 15465 tablet_copy_client.cc:647] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:39.944412 15465 tablet_copy_client.cc:523] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:39.948889 15021 tablet_copy_service.cc:342] P 9fd6981a1f214470ab279f88136f1b14: Request end of tablet copy session a88d3cb5c6774a43acf84bda9c2362e1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.168.125@KRBTEST.COM'} at 127.0.0.1:33402
I20240501 19:56:39.949000 15021 tablet_copy_service.cc:433] P 9fd6981a1f214470ab279f88136f1b14: ending tablet copy session a88d3cb5c6774a43acf84bda9c2362e1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a88d3cb5c6774a43acf84bda9c2362e1
I20240501 19:56:39.958480 15465 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:39.960439 15578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:39.961014 15579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:39.961593 15581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:39.961972 15465 server_base.cc:1034] running on GCE node
I20240501 19:56:39.962208 15465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:39.962527 15465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:39.963717 15465 hybrid_clock.cc:648] HybridClock initialized: now 1714593399963704 us; error 29 us; skew 500 ppm
May 01 19:56:39 dist-test-slave-tzct krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593399, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:39.965442 15465 init.cc:377] Logged in from keytab as oryx/127.8.168.125@KRBTEST.COM (short username oryx)
I20240501 19:56:39.965968 15465 webserver.cc:460] Webserver started at http://127.8.168.125:34039/ using document root <none> and password file <none>
I20240501 19:56:39.966200 15465 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:39.966292 15465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:39.976989 15465 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.001s sys 0.000s
I20240501 19:56:39.978677 15588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:39.984869 15465 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20240501 19:56:39.985042 15465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/wal
uuid: "a88d3cb5c6774a43acf84bda9c2362e1"
format_stamp: "Formatted at 2024-05-01 19:56:39 on dist-test-slave-tzct"
server_key: "017038339f375249ba4b25c6e4c47338"
server_key_iv: "24b7085893682540a1bd4d5c0643b9c3"
server_key_version: "encryptionkey@0"
I20240501 19:56:39.985330 15465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:40.005086 15465 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:40.006963 15465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:40.011102 15465 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.125:36409
I20240501 19:56:40.011127 15640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.125:36409 every 8 connection(s)
I20240501 19:56:40.011709 15465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-1/data/info.pb
I20240501 19:56:40.013609 15641 sys_catalog.cc:263] Verifying existing consensus state
I20240501 19:56:40.014330 15641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: Bootstrap starting.
I20240501 19:56:40.021848 15641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:40.022132 15641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: Bootstrap complete.
I20240501 19:56:40.022634 15641 raft_consensus.cc:348] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: NON_VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: true } }
I20240501 19:56:40.022747 15641 raft_consensus.cc:729] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a88d3cb5c6774a43acf84bda9c2362e1, State: Initialized, Role: LEARNER
I20240501 19:56:40.022843 15641 consensus_queue.cc:260] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: NON_VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: true } }
I20240501 19:56:40.023171 15644 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: NON_VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: true } } }
I20240501 19:56:40.023311 15641 sys_catalog.cc:564] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:40.023331 15644 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: This master's current role is: LEARNER
I20240501 19:56:40.026259 15655 catalog_manager.cc:1259] Loaded cluster ID: cb2d94b3f8bd46e9bbe19926806e1c00
I20240501 19:56:40.026373 15655 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: loading cluster ID for follower catalog manager: success
I20240501 19:56:40.027997 15655 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: acquiring CA information for follower catalog manager: success
I20240501 19:56:40.029109 15655 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20240501 19:56:40.076313 15568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:40.076642 15568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:40.076750 15568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:40.080857 15568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:40.080941 15568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:40.080972 15568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:40.080997 15568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:40.081022 15568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:40.085510 15568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:45289,127.8.168.125:36409,127.8.168.124:40517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.124.keytab
--principal=oryx/127.8.168.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.124:40517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.168.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:40.086409 15568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:40.087273 15568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:40.094331 15660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:40.094666 15663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:40.094872 15661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:40.095443 15568 server_base.cc:1034] running on GCE node
I20240501 19:56:40.095907 15568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:40.096802 15568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:40.097945 15568 hybrid_clock.cc:648] HybridClock initialized: now 1714593400097934 us; error 24 us; skew 500 ppm
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593400, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:40.100565 15568 init.cc:377] Logged in from keytab as oryx/127.8.168.124@KRBTEST.COM (short username oryx)
I20240501 19:56:40.101555 15568 webserver.cc:460] Webserver started at http://127.8.168.124:46501/ using document root <none> and password file <none>
I20240501 19:56:40.102053 15568 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:40.102100 15568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:40.102247 15568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:40.103979 15568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/data/instance:
uuid: "697109cc171e40c1bb4afb53014ca1bc"
format_stamp: "Formatted at 2024-05-01 19:56:40 on dist-test-slave-tzct"
server_key: "c38fd5697b0edabc9097b7c59805a707"
server_key_iv: "158babefd70707afd09ac644a7c30c07"
server_key_version: "encryptionkey@0"
I20240501 19:56:40.104408 15568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/wal/instance:
uuid: "697109cc171e40c1bb4afb53014ca1bc"
format_stamp: "Formatted at 2024-05-01 19:56:40 on dist-test-slave-tzct"
server_key: "c38fd5697b0edabc9097b7c59805a707"
server_key_iv: "158babefd70707afd09ac644a7c30c07"
server_key_version: "encryptionkey@0"
I20240501 19:56:40.107659 15568 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:40.109865 15670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:40.110862 15568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:40.110968 15568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/wal
uuid: "697109cc171e40c1bb4afb53014ca1bc"
format_stamp: "Formatted at 2024-05-01 19:56:40 on dist-test-slave-tzct"
server_key: "c38fd5697b0edabc9097b7c59805a707"
server_key_iv: "158babefd70707afd09ac644a7c30c07"
server_key_version: "encryptionkey@0"
I20240501 19:56:40.111070 15568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:40.146267 15568 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:40.149325 15568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:40.149495 15568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:40.156080 15568 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.124:40517
I20240501 19:56:40.156081 15722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.124:40517 every 8 connection(s)
I20240501 19:56:40.157634 15568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/data/info.pb
I20240501 19:56:40.159222 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 15568
I20240501 19:56:40.160966 15723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.165381 15723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } has no permanent_uuid. Determining permanent_uuid...
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.168.124@KRBTEST.COM
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593400, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.124@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM
I20240501 19:56:40.174185 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.161055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37446 (local address 127.8.168.124:40517)
0501 19:56:40.161635 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.161648 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.161731 (+ 83us) server_negotiation.cc:407] Connection header received
0501 19:56:40.162904 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.162931 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.163314 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.163721 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.164796 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.165708 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.166362 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.166622 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.169078 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.169145 (+ 67us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.169155 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.169200 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.171217 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.171640 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.171646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.171651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.171725 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.172253 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.172257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.172259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.172630 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.172839 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.173129 (+ 290us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.173381 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":141,"threads_started":1}
I20240501 19:56:40.174757 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.166868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.0.0.1:33404)
0501 19:56:40.167288 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.167334 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.167464 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.167746 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.167752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.168200 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:40.168426 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.168433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.169284 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.169287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.169648 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.169654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.169735 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.170200 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:40.170215 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:40.171733 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:40.173381 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.173387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.173390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.173618 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.173816 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.173819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.173821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.173869 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.174255 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:40.174260 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:40.174367 (+ 107us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.174508 (+ 141us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.174636 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":151,"threads_started":1}
I20240501 19:56:40.174875 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.167050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.8.168.126:45289)
0501 19:56:40.167185 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.167189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.167423 (+ 234us) server_negotiation.cc:407] Connection header received
0501 19:56:40.167560 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.167563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.167608 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.167676 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.168565 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.169158 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.169737 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.169932 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.171842 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.171861 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.171863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.171890 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.173273 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.173702 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.173705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.173706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.173747 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.173987 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.173990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.173992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.174145 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.174245 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.174604 (+ 359us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.174724 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20240501 19:56:40.176411 15723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } has no permanent_uuid. Determining permanent_uuid...
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593400, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.124@KRBTEST.COM for oryx/127.8.168.125@KRBTEST.COM
I20240501 19:56:40.184212 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.176934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36409 (local address 127.0.0.1:43148)
0501 19:56:40.177097 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.177109 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.177183 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.177502 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.177505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.177672 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:40.177887 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.177897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.179025 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.179029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.179419 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.179428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.179538 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.180011 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:40.180024 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:40.181314 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:40.182948 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.182952 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.182954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.183115 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.183327 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.183330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.183332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.183381 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.183713 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:40.183716 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:40.183802 (+ 86us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.183887 (+ 85us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.184023 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20240501 19:56:40.184214 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.177031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.8.168.125:36409)
0501 19:56:40.177292 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.177295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.177307 (+ 12us) server_negotiation.cc:407] Connection header received
0501 19:56:40.177343 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.177345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.177387 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.177479 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.178037 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.178905 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.179530 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.179726 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.181438 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.181458 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.181460 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.181485 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.182828 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.183197 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.183201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.183203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.183246 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.183474 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.183477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.183479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.183627 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.183716 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.183951 (+ 235us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.184070 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":83,"threads_started":1}
I20240501 19:56:40.185110 15723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } has no permanent_uuid. Determining permanent_uuid...
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593400, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.124@KRBTEST.COM for oryx/127.8.168.124@KRBTEST.COM
I20240501 19:56:40.192025 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.185591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:40517 (local address 127.0.0.1:37452)
0501 19:56:40.185743 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.185757 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.185829 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.186022 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.186025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.186162 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:40.186366 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.186373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.187033 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.187035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.187768 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.187776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.187865 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.188343 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:40.188356 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:40.189505 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:40.190884 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.190886 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.190888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.191057 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.191265 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.191268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.191269 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.191318 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.191599 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:40.191601 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:40.191671 (+ 70us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.191748 (+ 77us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.191847 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20240501 19:56:40.192034 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.185762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37452 (local address 127.8.168.124:40517)
0501 19:56:40.185861 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.185864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.185875 (+ 11us) server_negotiation.cc:407] Connection header received
0501 19:56:40.185902 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.185904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.185939 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.186002 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.186458 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.186955 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.187852 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.187990 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.189627 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.189642 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.189644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.189665 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.190787 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.191130 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.191132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.191133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.191168 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.191400 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.191402 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.191403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.191535 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.191599 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.191785 (+ 186us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.191901 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":34}
I20240501 19:56:40.194805 15723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: Bootstrap starting.
I20240501 19:56:40.196921 15723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.197614 15723 log.cc:824] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:40.199412 15723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: No bootstrap required, opened a new log
I20240501 19:56:40.202153 15723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } }
I20240501 19:56:40.202327 15723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.202395 15723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 697109cc171e40c1bb4afb53014ca1bc, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.202787 15723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } }
I20240501 19:56:40.203238 15729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } } }
I20240501 19:56:40.203341 15729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:56:40.203806 15723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [sys.catalog]: configured and running, proceeding with master startup.
W20240501 19:56:40.207793 15740 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:56:40.207873 15740 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:56:40.211287 15568 master_runner.cc:428] Detected that this master 697109cc171e40c1bb4afb53014ca1bc is joining an existing cluster
I20240501 19:56:40.211345 15568 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.168.124:40517
I20240501 19:56:40.227581 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 15063
I20240501 19:56:40.232892 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:46591
--local_ip_for_outbound_sockets=127.8.168.65
--tserver_master_addrs=127.8.168.126:45289
--webserver_port=38405
--webserver_interface=127.8.168.65
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.65.keytab
--principal=oryx/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.168.126:45289,127.8.168.125:36409
--tserver_master_addrs=127.8.168.126:45289,127.8.168.125:36409,127.8.168.124:40517 with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 19:56:40.253100 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.213489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.8.168.126:45289)
0501 19:56:40.213636 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.213640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.214150 (+ 510us) server_negotiation.cc:407] Connection header received
0501 19:56:40.214299 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.214303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.214354 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.214407 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.245811 (+ 31404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.246616 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.247547 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.247738 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.249735 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.249756 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.249759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.249790 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.251334 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.251790 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.251794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.251796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.251849 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.252079 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.252082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.252083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.252250 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.252316 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.252670 (+ 354us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.252930 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20240501 19:56:40.254398 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.213409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.0.0.1:33418)
0501 19:56:40.214061 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.214075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.214180 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.214555 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.214558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.245123 (+ 30565us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:40.245590 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.245604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.246788 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.246791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.247395 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.247405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.247878 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.249127 (+ 1249us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:40.249144 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:40.249591 (+ 447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:40.251455 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.251458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.251460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.251684 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.251943 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.251946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.251948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.251997 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.252378 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:40.252381 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:40.252459 (+ 78us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.253612 (+ 1153us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.254216 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"spinlock_wait_cycles":18432,"thread_start_us":119,"threads_started":1}
I20240501 19:56:40.257661 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.213816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:40517 (local address 127.0.0.1:37458)
0501 19:56:40.216976 (+ 3160us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.216992 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.217112 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.217345 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.217348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.246745 (+ 29397us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:40.247038 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.247046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.247916 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.247919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.249275 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.249286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.249394 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.251889 (+ 2495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:40.251904 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:40.252259 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:40.255009 (+ 2750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.255013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.255015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.255202 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.255774 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.255777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.255780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.255822 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.257238 (+ 1416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:40.257241 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:40.257302 (+ 61us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.257378 (+ 76us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.257498 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3073,"thread_start_us":81,"threads_started":1}
I20240501 19:56:40.258078 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.213976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.8.168.124:40517)
0501 19:56:40.214072 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.214076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.217084 (+ 3008us) server_negotiation.cc:407] Connection header received
0501 19:56:40.217196 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.217199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.217239 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.217303 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.247153 (+ 29850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.247814 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.249765 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.249950 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.252373 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.252388 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.252391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.252410 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.254096 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.255622 (+ 1526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.255626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.255628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.255675 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.256638 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.256642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.256644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.257135 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.257209 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.257857 (+ 648us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.257957 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20240501 19:56:40.260049 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.213484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36409 (local address 127.0.0.1:43152)
0501 19:56:40.214159 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.214172 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.214270 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.214573 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.214576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.245152 (+ 30576us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:40.245590 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.245604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.248452 (+ 2848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.248456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.250918 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.250928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.251031 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.253933 (+ 2902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:40.253949 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:40.254423 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:40.256772 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.256775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.256777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.256964 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.259151 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.259154 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.259156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.259207 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.259574 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:40.259577 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:40.259673 (+ 96us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.259773 (+ 100us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.259888 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"spinlock_wait_cycles":6016,"thread_start_us":94,"threads_started":1}
I20240501 19:56:40.260648 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.213772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43152 (local address 127.8.168.125:36409)
0501 19:56:40.213895 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.213899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.214248 (+ 349us) server_negotiation.cc:407] Connection header received
0501 19:56:40.214413 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.214417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.214463 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.214546 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.245808 (+ 31262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.246605 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.251352 (+ 4747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.251580 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.254584 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.254606 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.254610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.254639 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.256511 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.258990 (+ 2479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.258994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.258996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.259050 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.259290 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.259294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.259296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.259455 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.259557 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.260356 (+ 799us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.260484 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593398, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.126@KRBTEST.COM for oryx/127.8.168.124@KRBTEST.COM
I20240501 19:56:40.267470 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.259274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:40517 (local address 127.0.0.1:37466)
0501 19:56:40.259669 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.259683 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.259751 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.260273 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.260276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.260455 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:40.260703 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.260710 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.261637 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.261641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.262198 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.262208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.262291 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.262826 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:40.262842 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:40.264290 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:40.266162 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.266165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.266167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.266367 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.266598 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.266601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.266602 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.266655 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.266981 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:40.266984 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:40.267076 (+ 92us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.267177 (+ 101us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.267309 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306}
I20240501 19:56:40.267498 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.259874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.8.168.124:40517)
0501 19:56:40.259975 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.259979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.259990 (+ 11us) server_negotiation.cc:407] Connection header received
0501 19:56:40.260051 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.260054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.260101 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.260167 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.260844 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.261518 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.262524 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.262708 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.264409 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.264426 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.264430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.264454 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.266046 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.266460 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.266464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.266472 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.266515 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.266732 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.266735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.266736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.266896 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.266988 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.267229 (+ 241us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.267335 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240501 19:56:40.268568 15001 catalog_manager.cc:6968] Initiating ChangeConfig request to add master 127.8.168.124:40517: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc"
member_type: NON_VOTER
last_known_addr {
host: "127.8.168.124"
port: 40517
}
attrs {
promote: true
}
}
dest_uuid: "9fd6981a1f214470ab279f88136f1b14"
cas_config_opid_index: 5
I20240501 19:56:40.269168 15001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: NON_VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: true } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: NON_VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: true } }
I20240501 19:56:40.270255 15563 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 697109cc171e40c1bb4afb53014ca1bc (127.8.168.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: NON_VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: true } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: NON_VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: true } } }
I20240501 19:56:40.271647 15563 catalog_manager.cc:6972] Successfully completed master ChangeConfig request to add master 127.8.168.124:40517
I20240501 19:56:40.271831 15563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9fd6981a1f214470ab279f88136f1b14" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: NON_VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: true } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: NON_VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: true } } }
I20240501 19:56:40.271926 15563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:40.272356 15568 master.cc:537] Master@127.8.168.124:40517 shutting down...
W20240501 19:56:40.274310 14986 consensus_peers.cc:485] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 -> Peer 697109cc171e40c1bb4afb53014ca1bc (127.8.168.124:40517): Couldn't send request to peer 697109cc171e40c1bb4afb53014ca1bc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:40.277595 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.272690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43154 (local address 127.8.168.125:36409)
0501 19:56:40.273400 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.273404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.273685 (+ 281us) server_negotiation.cc:407] Connection header received
0501 19:56:40.273740 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.273744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.273804 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.273909 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:40.274358 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.275276 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.276797 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.277233 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.277362 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.277416 (+ 54us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.277471 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616}
I20240501 19:56:40.277953 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.272431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36409 (local address 127.0.0.1:43154)
0501 19:56:40.273499 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.273513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.273602 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.273962 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.273965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.273991 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:40.274218 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.274225 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.275409 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.275413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.276311 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.276323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.276435 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.276501 (+ 66us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.276560 (+ 59us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.276629 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989}
I20240501 19:56:40.278955 15611 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 1 LEARNER]: Refusing update from remote peer 9fd6981a1f214470ab279f88136f1b14: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:40.279445 15562 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: NON_VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:40.279992 15568 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.280256 15568 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.280340 15568 tablet_replica.cc:330] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: stopping tablet replica
I20240501 19:56:40.281419 15611 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 697109cc171e40c1bb4afb53014ca1bc (127.8.168.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: NON_VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: true } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: NON_VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: true } } }
I20240501 19:56:40.283565 15644 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fd6981a1f214470ab279f88136f1b14. Latest consensus state: current_term: 1 leader_uuid: "9fd6981a1f214470ab279f88136f1b14" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: NON_VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: true } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: NON_VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: true } } }
I20240501 19:56:40.283689 15644 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: This master's current role is: LEARNER
I20240501 19:56:40.283788 15644 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fd6981a1f214470ab279f88136f1b14" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: NON_VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: true } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: NON_VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: true } } }
I20240501 19:56:40.283855 15644 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: This master's current role is: LEARNER
I20240501 19:56:40.290263 15562 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14: attempting to promote NON_VOTER a88d3cb5c6774a43acf84bda9c2362e1 to VOTER
I20240501 19:56:40.290746 15562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: NON_VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: true } }
I20240501 19:56:40.292546 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.292233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:40517 (local address 127.0.0.1:37472)
0501 19:56:40.292372 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.292447 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:40517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20240501 19:56:40.292779 15611 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 1 LEARNER]: Refusing update from remote peer 9fd6981a1f214470ab279f88136f1b14: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:40.293371 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.293156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:40517 (local address 127.0.0.1:37486)
0501 19:56:40.293254 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.293309 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:40517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20240501 19:56:40.293356 15561 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 19:56:40.293843 14986 consensus_peers.cc:485] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 -> Peer 697109cc171e40c1bb4afb53014ca1bc (127.8.168.124:40517): Couldn't send request to peer 697109cc171e40c1bb4afb53014ca1bc. Status: Network error: Client connection negotiation failed: client connection to 127.8.168.124:40517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:40.295460 15563 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a88d3cb5c6774a43acf84bda9c2362e1 (127.8.168.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: NON_VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: true } } }
I20240501 19:56:40.295670 15568 master.cc:552] Master@127.8.168.124:40517 shutdown complete.
I20240501 19:56:40.295718 15568 master_runner.cc:305] Clearing existing system tablet
I20240501 19:56:40.295935 15568 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:40.296000 15568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:40.296862 15563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9fd6981a1f214470ab279f88136f1b14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: NON_VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: true } } }
I20240501 19:56:40.296967 15563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:40.297088 15563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fd6981a1f214470ab279f88136f1b14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: NON_VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: true } } }
I20240501 19:56:40.297149 15563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:40.297511 15568 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 19:56:40.297714 15611 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a88d3cb5c6774a43acf84bda9c2362e1 (127.8.168.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: NON_VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: true } } }
I20240501 19:56:40.299183 15751 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fd6981a1f214470ab279f88136f1b14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: NON_VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: true } } }
I20240501 19:56:40.299579 15751 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:56:40.301196 15753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:40.301401 15568 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:40.301527 15568 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:40.301571 15568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/wal
uuid: "697109cc171e40c1bb4afb53014ca1bc"
format_stamp: "Formatted at 2024-05-01 19:56:40 on dist-test-slave-tzct"
server_key: "c38fd5697b0edabc9097b7c59805a707"
server_key_iv: "158babefd70707afd09ac644a7c30c07"
server_key_version: "encryptionkey@0"
I20240501 19:56:40.303526 15568 ts_tablet_manager.cc:1875] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:40.305848 15568 ts_tablet_manager.cc:1888] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20240501 19:56:40.305936 15568 log.cc:1196] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20240501 19:56:40.306150 15568 ts_tablet_manager.cc:1909] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: Deleting consensus metadata
I20240501 19:56:40.306313 15568 master_runner.cc:315] Copying system tablet from 127.8.168.126:45289
I20240501 19:56:40.307622 15568 tablet_copy_client.cc:308] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: tablet copy: Beginning tablet copy session from remote peer at address 127.8.168.126:45289
I20240501 19:56:40.316334 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.308256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33432 (local address 127.8.168.126:45289)
0501 19:56:40.308387 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.308391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.308632 (+ 241us) server_negotiation.cc:407] Connection header received
0501 19:56:40.308871 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.308875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.308928 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.308979 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.309795 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.310571 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.311473 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.311663 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.313038 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.313059 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.313063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.313094 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.314698 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.315175 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.315179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.315181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.315230 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.315476 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.315479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.315481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.315665 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.315733 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.316056 (+ 323us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.316170 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240501 19:56:40.316874 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.308153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.0.0.1:33432)
0501 19:56:40.308530 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.308555 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.308712 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.309123 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.309126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.309321 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:40.309629 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.309638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.310712 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.310715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.311314 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.311325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.311794 (+ 469us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.312388 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:40.312405 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:40.312903 (+ 498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:40.314843 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.314846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.314848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.315066 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.315327 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.315329 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.315331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.315385 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.315799 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:40.315802 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:40.315888 (+ 86us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.316534 (+ 646us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.316679 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":87,"threads_started":1}
I20240501 19:56:40.317426 15021 tablet_copy_service.cc:140] P 9fd6981a1f214470ab279f88136f1b14: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 697109cc171e40c1bb4afb53014ca1bc ({username='oryx', principal='oryx/127.8.168.124@KRBTEST.COM'} at 127.0.0.1:33432)
I20240501 19:56:40.317538 15021 tablet_copy_service.cc:161] P 9fd6981a1f214470ab279f88136f1b14: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 697109cc171e40c1bb4afb53014ca1bc at {username='oryx', principal='oryx/127.8.168.124@KRBTEST.COM'} at 127.0.0.1:33432: session id = 697109cc171e40c1bb4afb53014ca1bc-00000000000000000000000000000000
I20240501 19:56:40.318574 15021 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:40.319797 15568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20240501 19:56:40.322052 15568 tablet_copy_client.cc:783] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:40.322221 15568 tablet_copy_client.cc:647] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:40.323851 15568 tablet_copy_client.cc:523] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:40.325778 15021 tablet_copy_service.cc:342] P 9fd6981a1f214470ab279f88136f1b14: Request end of tablet copy session 697109cc171e40c1bb4afb53014ca1bc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.168.124@KRBTEST.COM'} at 127.0.0.1:33432
I20240501 19:56:40.325876 15021 tablet_copy_service.cc:433] P 9fd6981a1f214470ab279f88136f1b14: ending tablet copy session 697109cc171e40c1bb4afb53014ca1bc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 697109cc171e40c1bb4afb53014ca1bc
I20240501 19:56:40.333412 15568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:40.335333 15763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:40.335901 15766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:40.336561 15568 server_base.cc:1034] running on GCE node
W20240501 19:56:40.336897 15764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:40.337152 15568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:40.337493 15568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:40.338629 15568 hybrid_clock.cc:648] HybridClock initialized: now 1714593400338617 us; error 33 us; skew 500 ppm
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593400, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:40.340297 15568 init.cc:377] Logged in from keytab as oryx/127.8.168.124@KRBTEST.COM (short username oryx)
I20240501 19:56:40.340823 15568 webserver.cc:460] Webserver started at http://127.8.168.124:45593/ using document root <none> and password file <none>
I20240501 19:56:40.341010 15568 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:40.341049 15568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:40.342348 15568 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:40.343361 15773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:40.343669 15568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:40.343745 15568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/wal
uuid: "697109cc171e40c1bb4afb53014ca1bc"
format_stamp: "Formatted at 2024-05-01 19:56:40 on dist-test-slave-tzct"
server_key: "c38fd5697b0edabc9097b7c59805a707"
server_key_iv: "158babefd70707afd09ac644a7c30c07"
server_key_version: "encryptionkey@0"
I20240501 19:56:40.343828 15568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240501 19:56:40.368208 15749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:40.368448 15749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:40.368527 15749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:40.371701 15749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:40.371768 15749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:40.371872 15749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:56:40.376093 15749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.65.keytab
--principal=oryx/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:46591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=38405
--tserver_master_addrs=127.8.168.126:45289,127.8.168.125:36409,127.8.168.124:40517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:40.377091 15749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:40.377759 15568 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:40.377969 15749 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:56:40.380168 15568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:40.384853 15568 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.124:40517
I20240501 19:56:40.384872 15833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.124:40517 every 8 connection(s)
I20240501 19:56:40.385540 15568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/master-2/data/info.pb
W20240501 19:56:40.387703 15819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:40.387771 15823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:40.387523 15834 sys_catalog.cc:263] Verifying existing consensus state
W20240501 19:56:40.388339 15816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:40.388943 15749 server_base.cc:1034] running on GCE node
I20240501 19:56:40.388998 15834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: Bootstrap starting.
I20240501 19:56:40.389391 15749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:40.390079 15749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:40.391240 15749 hybrid_clock.cc:648] HybridClock initialized: now 1714593400391210 us; error 46 us; skew 500 ppm
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593400, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:40.394207 15749 init.cc:377] Logged in from keytab as oryx/127.8.168.65@KRBTEST.COM (short username oryx)
I20240501 19:56:40.395154 15749 webserver.cc:460] Webserver started at http://127.8.168.65:38405/ using document root <none> and password file <none>
I20240501 19:56:40.395663 15749 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:40.395731 15749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:40.398646 15834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:40.398914 15834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: Bootstrap complete.
I20240501 19:56:40.399148 15749 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:40.399475 15834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: NON_VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: true } }
I20240501 19:56:40.399608 15834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 697109cc171e40c1bb4afb53014ca1bc, State: Initialized, Role: LEARNER
I20240501 19:56:40.399814 15834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: NON_VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: true } }
I20240501 19:56:40.400183 15843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: NON_VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: true } } }
I20240501 19:56:40.400341 15834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:40.400357 15843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [sys.catalog]: This master's current role is: LEARNER
I20240501 19:56:40.401384 15844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:40.402631 15749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:40.402743 15749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "3247c2f74cd44a12a587f056979b798f"
format_stamp: "Formatted at 2024-05-01 19:56:39 on dist-test-slave-tzct"
server_key: "6e7d07928a6f4fa7cd182ab7565c5179"
server_key_iv: "55d8889cf3a869e43e5bc6a65bf1048a"
server_key_version: "encryptionkey@0"
I20240501 19:56:40.403352 15855 catalog_manager.cc:1259] Loaded cluster ID: cb2d94b3f8bd46e9bbe19926806e1c00
I20240501 19:56:40.403281 15749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:40.403385 15855 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: loading cluster ID for follower catalog manager: success
I20240501 19:56:40.405591 15855 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: acquiring CA information for follower catalog manager: success
I20240501 19:56:40.407070 15855 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:56:40.422735 15749 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:40.425926 15749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:40.426095 15749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:40.426632 15749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:40.427455 15749 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:40.427501 15749 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:40.427542 15749 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:40.427569 15749 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:40.435580 15749 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:46591
I20240501 19:56:40.435613 15968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:46591 every 8 connection(s)
I20240501 19:56:40.436597 15749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 19:56:40.442024 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 15749
I20240501 19:56:40.442215 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 15197
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593400, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593400, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.125@KRBTEST.COM
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593400, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.124@KRBTEST.COM
I20240501 19:56:40.450276 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:41839
--local_ip_for_outbound_sockets=127.8.168.66
--tserver_master_addrs=127.8.168.126:45289
--webserver_port=36205
--webserver_interface=127.8.168.66
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.66.keytab
--principal=oryx/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.168.126:45289,127.8.168.125:36409
--tserver_master_addrs=127.8.168.126:45289,127.8.168.125:36409,127.8.168.124:40517 with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 19:56:40.451109 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.438272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:42601 (local address 127.8.168.126:45289)
0501 19:56:40.438398 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.438401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.439243 (+ 842us) server_negotiation.cc:407] Connection header received
0501 19:56:40.440224 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.440227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.440278 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.440365 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.441979 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.442767 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.444041 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.444237 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.446917 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.446936 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.446939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.446968 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.448607 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.449329 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.449333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.449334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.449387 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.449669 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.449672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.449673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.449844 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.449933 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.450835 (+ 902us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.450950 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240501 19:56:40.451875 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.438639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36409 (local address 127.8.168.65:44111)
0501 19:56:40.439539 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.439552 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.440016 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.440904 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.440907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.441166 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:40.441837 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.441846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.442894 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.442898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.443508 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.443517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.443726 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.444468 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:40.444492 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:40.446705 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:40.448878 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.448881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.448883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.449196 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.449446 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.449449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.449451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.449568 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.449986 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:40.449992 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:40.450294 (+ 302us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.450532 (+ 238us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.450836 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":39,"spinlock_wait_cycles":42624,"thread_start_us":121,"threads_started":1}
I20240501 19:56:40.452543 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.438768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:44111 (local address 127.8.168.125:36409)
0501 19:56:40.438899 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.438904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.439620 (+ 716us) server_negotiation.cc:407] Connection header received
0501 19:56:40.440218 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.440222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.440278 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.440365 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.441984 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.442756 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.444041 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.444237 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.446894 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.446918 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.446921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.446951 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.448771 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.449298 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.449302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.449304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.449353 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.449643 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.449646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.449648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.449834 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.449927 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.452257 (+ 2330us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.452388 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 19:56:40.452960 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.438639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.65:42601)
0501 19:56:40.439088 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.439132 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.440004 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.440552 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.440563 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.441161 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:40.441842 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.441847 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.442919 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.442922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.443508 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.443517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.443727 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.444570 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:40.444583 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:40.446787 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:40.448755 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.448761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.448773 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.449191 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.449477 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.449480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.449482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.449576 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.450257 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:40.450260 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:40.450351 (+ 91us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.450530 (+ 179us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.452802 (+ 2272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":1959,"spinlock_wait_cycles":81408,"thread_start_us":123,"threads_started":1}
I20240501 19:56:40.453585 15971 heartbeater.cc:347] Connected to a master server at 127.8.168.125:36409
I20240501 19:56:40.453593 15970 heartbeater.cc:347] Connected to a master server at 127.8.168.126:45289
I20240501 19:56:40.453852 15970 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:40.453852 15971 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:40.454501 15971 heartbeater.cc:511] Master 127.8.168.125:36409 requested a full tablet report, sending...
I20240501 19:56:40.454501 15970 heartbeater.cc:511] Master 127.8.168.126:45289 requested a full tablet report, sending...
I20240501 19:56:40.455837 15001 ts_manager.cc:194] Re-registered known tserver with Master: 3247c2f74cd44a12a587f056979b798f (127.8.168.65:46591)
I20240501 19:56:40.456068 15976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.439523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:40517 (local address 127.8.168.65:43375)
0501 19:56:40.440396 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.440408 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.440487 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.440802 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.440804 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.441160 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:40.441784 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.441802 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.443207 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.443210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.443841 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.443882 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.443980 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.445153 (+ 1173us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:40.445169 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:40.447351 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:40.451498 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.451502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.451504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.451712 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.452200 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.452203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.452205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.452262 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.455608 (+ 3346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:40.455611 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:40.455705 (+ 94us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.455783 (+ 78us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.455918 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"spinlock_wait_cycles":4864,"thread_start_us":487,"threads_started":1}
I20240501 19:56:40.456140 15605 ts_manager.cc:194] Registered new tserver with Master: 3247c2f74cd44a12a587f056979b798f (127.8.168.65:46591)
I20240501 19:56:40.456605 15001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:42601
I20240501 19:56:40.456833 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.439644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:43375 (local address 127.8.168.124:40517)
0501 19:56:40.439912 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.439916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.440561 (+ 645us) server_negotiation.cc:407] Connection header received
0501 19:56:40.440603 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.440606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.440655 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.440778 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.441949 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.443078 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.445695 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.445905 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.447465 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.447486 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.447490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.447519 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.451376 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.451816 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.451820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.451823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.451882 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.452369 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.452372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.452374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.452550 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.456321 (+ 3771us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.456400 (+ 79us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.456645 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
I20240501 19:56:40.457437 15969 heartbeater.cc:347] Connected to a master server at 127.8.168.124:40517
I20240501 19:56:40.457535 15969 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:40.457873 15969 heartbeater.cc:511] Master 127.8.168.124:40517 requested a full tablet report, sending...
I20240501 19:56:40.458909 15793 ts_manager.cc:194] Registered new tserver with Master: 3247c2f74cd44a12a587f056979b798f (127.8.168.65:46591)
W20240501 19:56:40.563468 15977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:40.563719 15977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:40.563800 15977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:40.566856 15977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:40.566932 15977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:40.567040 15977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:56:40.571151 15977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.66.keytab
--principal=oryx/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:41839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=36205
--tserver_master_addrs=127.8.168.126:45289,127.8.168.125:36409,127.8.168.124:40517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:40.572111 15977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:40.573017 15977 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:40.580834 15983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:40.581350 15985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:40.581955 15982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:40.582477 15977 server_base.cc:1034] running on GCE node
I20240501 19:56:40.582851 15977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:40.583392 15977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:40.584517 15977 hybrid_clock.cc:648] HybridClock initialized: now 1714593400584495 us; error 31 us; skew 500 ppm
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593400, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:40.587426 15977 init.cc:377] Logged in from keytab as oryx/127.8.168.66@KRBTEST.COM (short username oryx)
I20240501 19:56:40.588582 15977 webserver.cc:460] Webserver started at http://127.8.168.66:36205/ using document root <none> and password file <none>
I20240501 19:56:40.589295 15977 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:40.589383 15977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:40.592878 15977 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20240501 19:56:40.594733 15992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:40.595713 15977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:40.595808 15977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "2e74f065d34341d8a34c0b20761a558f"
format_stamp: "Formatted at 2024-05-01 19:56:39 on dist-test-slave-tzct"
server_key: "99764d94e2c5e51ff779af19e4a9c520"
server_key_iv: "1df0af08a762b99f081dffa88fc7d473"
server_key_version: "encryptionkey@0"
I20240501 19:56:40.596207 15977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:40.611172 15977 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:40.614005 15977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:40.614154 15977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:40.614660 15977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:40.615455 15977 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:40.615502 15977 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:40.615540 15977 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:40.615552 15977 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:40.624903 15977 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:41839
I20240501 19:56:40.624920 16105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:41839 every 8 connection(s)
I20240501 19:56:40.625911 15977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 19:56:40.627491 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 15977
I20240501 19:56:40.627691 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 15331
I20240501 19:56:40.635129 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:37273
--local_ip_for_outbound_sockets=127.8.168.67
--tserver_master_addrs=127.8.168.126:45289
--webserver_port=42119
--webserver_interface=127.8.168.67
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.67.keytab
--principal=oryx/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.168.126:45289,127.8.168.125:36409
--tserver_master_addrs=127.8.168.126:45289,127.8.168.125:36409,127.8.168.124:40517 with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593400, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.124@KRBTEST.COM
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593400, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.125@KRBTEST.COM
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593400, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM
I20240501 19:56:40.642201 16112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.627829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:40517 (local address 127.8.168.66:43987)
0501 19:56:40.629121 (+ 1292us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.629164 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.630525 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.631032 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.631043 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.631550 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:40.632317 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.632334 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.633113 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.633116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.633564 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.633573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.633780 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.634395 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:40.634420 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:40.636155 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:40.639285 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.639291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.639305 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.639644 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.639902 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.639904 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.639906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.640029 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.640435 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:40.640441 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:40.640771 (+ 330us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.640998 (+ 227us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.641301 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1103,"thread_start_us":99,"threads_started":1}
I20240501 19:56:40.643003 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.627698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:43987 (local address 127.8.168.124:40517)
0501 19:56:40.627958 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.627962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.629263 (+ 1301us) server_negotiation.cc:407] Connection header received
0501 19:56:40.630736 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.630739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.630787 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.630854 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.632459 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.632984 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.633682 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.633880 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.636321 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.636342 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.636346 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.636374 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.639128 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.639754 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.639757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.639759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.639812 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.640125 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.640128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.640129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.640291 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.640376 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.642718 (+ 2342us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.642850 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20240501 19:56:40.643003 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.627567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:50057 (local address 127.8.168.126:45289)
0501 19:56:40.627829 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.627833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.634244 (+ 6411us) server_negotiation.cc:407] Connection header received
0501 19:56:40.635146 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.635150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.635207 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.635259 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.635978 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.636695 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.637511 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.637690 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.639820 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.639841 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.639845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.639875 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.641468 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.641872 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.641876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.641878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.641924 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.642168 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.642171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.642173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.642344 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.642417 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.642739 (+ 322us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.642848 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20240501 19:56:40.643690 16111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.627829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.66:50057)
0501 19:56:40.634159 (+ 6330us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.634172 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.635005 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.635403 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.635406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.635575 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:40.635839 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.635848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.636855 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.636858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.637388 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.637400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.637513 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.637950 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:40.637966 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:40.639701 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:40.641585 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.641588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.641590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.641771 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.642018 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.642021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.642022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.642082 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.642482 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:40.642485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:40.642562 (+ 77us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.643374 (+ 812us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.643512 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6243,"mutex_wait_us":914,"thread_start_us":97,"threads_started":1}
I20240501 19:56:40.644006 16106 heartbeater.cc:347] Connected to a master server at 127.8.168.124:40517
I20240501 19:56:40.644070 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.628815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:47791 (local address 127.8.168.125:36409)
0501 19:56:40.629261 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.629265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.633411 (+ 4146us) server_negotiation.cc:407] Connection header received
0501 19:56:40.633513 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.633517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.633573 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.633657 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.634596 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.635248 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.636025 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.636195 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.639525 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.639548 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.639552 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.639585 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.641727 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.642190 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.642194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.642196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.642252 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.642785 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.642788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.642790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.642980 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.643056 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.643771 (+ 715us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.643894 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364}
I20240501 19:56:40.644258 16106 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:40.644358 16107 heartbeater.cc:347] Connected to a master server at 127.8.168.126:45289
I20240501 19:56:40.644447 16107 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:40.644618 16110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.628652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36409 (local address 127.8.168.66:47791)
0501 19:56:40.633313 (+ 4661us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.633327 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.633417 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.634002 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.634006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.634191 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:40.634466 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.634475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.635383 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.635386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.635900 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.635908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.636315 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.636918 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:40.636934 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:40.639391 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:40.641865 (+ 2474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.641869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.641871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.642073 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.642618 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.642620 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.642622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.642680 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.643498 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:40.643501 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:40.643588 (+ 87us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.644239 (+ 651us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.644405 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4584,"thread_start_us":94,"threads_started":1}
I20240501 19:56:40.644903 16107 heartbeater.cc:511] Master 127.8.168.126:45289 requested a full tablet report, sending...
I20240501 19:56:40.644903 16106 heartbeater.cc:511] Master 127.8.168.124:40517 requested a full tablet report, sending...
I20240501 19:56:40.645454 16108 heartbeater.cc:347] Connected to a master server at 127.8.168.125:36409
I20240501 19:56:40.645517 16108 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:40.645705 16108 heartbeater.cc:511] Master 127.8.168.125:36409 requested a full tablet report, sending...
I20240501 19:56:40.645994 15001 ts_manager.cc:194] Re-registered known tserver with Master: 2e74f065d34341d8a34c0b20761a558f (127.8.168.66:41839)
I20240501 19:56:40.646060 15793 ts_manager.cc:194] Registered new tserver with Master: 2e74f065d34341d8a34c0b20761a558f (127.8.168.66:41839)
I20240501 19:56:40.646488 15605 ts_manager.cc:194] Registered new tserver with Master: 2e74f065d34341d8a34c0b20761a558f (127.8.168.66:41839)
I20240501 19:56:40.646639 15001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:50057
W20240501 19:56:40.747737 16113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:40.747995 16113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:40.748064 16113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:40.751269 16113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:40.751339 16113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:40.751438 16113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:56:40.755589 16113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.67.keytab
--principal=oryx/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:37273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=42119
--tserver_master_addrs=127.8.168.126:45289,127.8.168.125:36409,127.8.168.124:40517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:40.756515 16113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:40.757390 16113 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:40.765990 16119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:40.766323 16118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:40.766525 16121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:40.767088 16113 server_base.cc:1034] running on GCE node
I20240501 19:56:40.767532 16113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:40.768209 16113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:40.769363 16113 hybrid_clock.cc:648] HybridClock initialized: now 1714593400769351 us; error 35 us; skew 500 ppm
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593400, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:40.772010 16113 init.cc:377] Logged in from keytab as oryx/127.8.168.67@KRBTEST.COM (short username oryx)
I20240501 19:56:40.773003 16113 webserver.cc:460] Webserver started at http://127.8.168.67:42119/ using document root <none> and password file <none>
I20240501 19:56:40.773530 16113 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:40.773602 16113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:40.777168 16113 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:40.779146 16128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:40.780238 16113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:40.780337 16113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "2522fdb3bfeb457e9a395935914b896c"
format_stamp: "Formatted at 2024-05-01 19:56:39 on dist-test-slave-tzct"
server_key: "b22d74993a5ff016a1c9f2b579dadf41"
server_key_iv: "1480aaa14336be89f6071f876a54dbaa"
server_key_version: "encryptionkey@0"
I20240501 19:56:40.780823 16113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:40.793694 16113 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:40.796418 16113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:40.796581 16113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:40.797161 16113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:40.798053 16113 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:40.798100 16113 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:40.798143 16113 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:40.798171 16113 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:40.806910 16113 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:37273
I20240501 19:56:40.806967 16241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:37273 every 8 connection(s)
I20240501 19:56:40.808080 16113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 19:56:40.811662 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 16113
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593400, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593400, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.125@KRBTEST.COM
May 01 19:56:40 dist-test-slave-tzct krb5kdc[14956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593400, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.124@KRBTEST.COM
I20240501 19:56:40.824096 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.809988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:48869 (local address 127.8.168.125:36409)
0501 19:56:40.810103 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.810107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.810787 (+ 680us) server_negotiation.cc:407] Connection header received
0501 19:56:40.812470 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.812474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.812527 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.812640 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.814550 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.815259 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.816126 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.816293 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.818741 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.818762 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.818765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.818795 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.821445 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.822733 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.822736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.822738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.822788 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.823045 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.823048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.823050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.823216 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.823752 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.823828 (+ 76us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.823937 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20240501 19:56:40.824371 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.811057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:40517 (local address 127.8.168.67:50061)
0501 19:56:40.811301 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.811313 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.812249 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.812818 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.812826 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.815217 (+ 2391us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:40.815458 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.815465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.817275 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.817278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.817620 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.817627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.817722 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.818200 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:40.818216 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:40.819581 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:40.821482 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.821489 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.821505 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.821849 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.822109 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.822112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.822115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.822233 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.822615 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:40.822621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:40.822955 (+ 334us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.823169 (+ 214us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.823474 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"spinlock_wait_cycles":4352,"thread_start_us":80,"threads_started":1}
I20240501 19:56:40.824656 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.809871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:36999 (local address 127.8.168.126:45289)
0501 19:56:40.810003 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.810008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.811137 (+ 1129us) server_negotiation.cc:407] Connection header received
0501 19:56:40.812204 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.812208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.812264 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.812343 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.814539 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.815042 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.816169 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.816356 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.818932 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.818950 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.818953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.818975 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.822621 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.823061 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.823064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.823066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.823115 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.823350 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.823353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.823355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.823515 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.823593 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.824404 (+ 811us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.824507 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20240501 19:56:40.824936 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.810215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.67:36999)
0501 19:56:40.811063 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.811074 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.811961 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.812819 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.812828 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.813908 (+ 1080us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:40.814396 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.814412 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.815158 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.815161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.815687 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.815695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.815879 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.816522 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:40.816555 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:40.818815 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:40.822770 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.822773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.822775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.822958 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.823209 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.823212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.823214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.823262 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.823631 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:40.823634 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:40.824003 (+ 369us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.824110 (+ 107us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.824227 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":25,"spinlock_wait_cycles":15232,"thread_start_us":109,"threads_started":1}
I20240501 19:56:40.825223 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.811168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:50061 (local address 127.8.168.124:40517)
0501 19:56:40.811286 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.811290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.811382 (+ 92us) server_negotiation.cc:407] Connection header received
0501 19:56:40.812358 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.812361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.812420 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.812501 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.815587 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.816880 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.817727 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.817907 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.819710 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.819730 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.819733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.819760 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.821336 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.821964 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.821968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.821970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.822019 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.822321 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.822324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.822326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.822487 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.822565 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.824967 (+ 2402us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.825083 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20240501 19:56:40.825857 16243 heartbeater.cc:347] Connected to a master server at 127.8.168.124:40517
I20240501 19:56:40.826318 16244 heartbeater.cc:347] Connected to a master server at 127.8.168.126:45289
I20240501 19:56:40.826385 16243 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:40.826398 16244 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:40.827009 16244 heartbeater.cc:511] Master 127.8.168.126:45289 requested a full tablet report, sending...
I20240501 19:56:40.827028 16243 heartbeater.cc:511] Master 127.8.168.124:40517 requested a full tablet report, sending...
I20240501 19:56:40.828142 15793 ts_manager.cc:194] Registered new tserver with Master: 2522fdb3bfeb457e9a395935914b896c (127.8.168.67:37273)
I20240501 19:56:40.828141 15001 ts_manager.cc:194] Re-registered known tserver with Master: 2522fdb3bfeb457e9a395935914b896c (127.8.168.67:37273)
I20240501 19:56:40.828912 15001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:36999
I20240501 19:56:40.826311 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.810215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36409 (local address 127.8.168.67:48869)
0501 19:56:40.810652 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.810695 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.811957 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.812972 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.812975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.813394 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:40.814395 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.814410 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.815398 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.815401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.816004 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.816013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.816400 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.816935 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:40.816952 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:40.818555 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:40.821792 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.821795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.821798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.821998 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.822896 (+ 898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:40.822898 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:40.822901 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:40.822951 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:40.823322 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:40.823325 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:40.823406 (+ 81us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.823490 (+ 84us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.823631 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":103,"threads_started":1}
I20240501 19:56:40.829748 16245 heartbeater.cc:347] Connected to a master server at 127.8.168.125:36409
I20240501 19:56:40.829824 16245 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:40.830035 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.813843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.8.168.124:40517)
0501 19:56:40.814075 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.814079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.820904 (+ 6825us) server_negotiation.cc:407] Connection header received
0501 19:56:40.821018 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.821022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.821088 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.821184 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.821978 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.822660 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.823709 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.823928 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.824923 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.824954 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.824957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.824976 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.827524 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.828121 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.828124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.828126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.828176 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.828417 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.828420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.828422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.828610 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.828761 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.829747 (+ 986us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.829863 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":76,"threads_started":1}
I20240501 19:56:40.830327 16245 heartbeater.cc:511] Master 127.8.168.125:36409 requested a full tablet report, sending...
I20240501 19:56:40.831019 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.813693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.8.168.126:45289)
0501 19:56:40.818841 (+ 5148us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.818844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.818857 (+ 13us) server_negotiation.cc:407] Connection header received
0501 19:56:40.818891 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.818894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.818953 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.819031 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.819864 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.820539 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.826027 (+ 5488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.826225 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.827540 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.827559 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.827562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.827586 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.829240 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.829656 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.829660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.829661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.829709 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.829941 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.829944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.829946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.830104 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.830224 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.830743 (+ 519us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.830863 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5064,"thread_start_us":62,"threads_started":1}
I20240501 19:56:40.831022 15605 ts_manager.cc:194] Registered new tserver with Master: 2522fdb3bfeb457e9a395935914b896c (127.8.168.67:37273)
I20240501 19:56:40.831506 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.814047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43158 (local address 127.8.168.125:36409)
0501 19:56:40.814274 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.814277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.814290 (+ 13us) server_negotiation.cc:407] Connection header received
0501 19:56:40.814322 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.814325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.814365 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.814475 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:40.815403 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.816129 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.817074 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.819880 (+ 2806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.819983 (+ 103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:40.819999 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:40.820002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:40.820021 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:40.825220 (+ 5199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.826009 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.826011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.826013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.826061 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:40.827826 (+ 1765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:40.827829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:40.827831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:40.829009 (+ 1178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:40.829143 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.830020 (+ 877us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.830223 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":66,"threads_started":1}
I20240501 19:56:40.833570 15001 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:56:40.835919 15001 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:40.850931 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.846990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:37273 (local address 127.0.0.1:57916)
0501 19:56:40.847247 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.847261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.847344 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.848470 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.848473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.848492 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:40.848781 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.848790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.849634 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.849638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.850539 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.850549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.850652 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.850673 (+ 21us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.850718 (+ 45us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.850776 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"spinlock_wait_cycles":3968,"thread_start_us":98,"threads_started":1}
I20240501 19:56:40.851444 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.847509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:46591 (local address 127.0.0.1:36226)
0501 19:56:40.848168 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.848180 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.848253 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.849047 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.849050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.849069 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:40.849288 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.849294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.850496 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.850499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.851093 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.851101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.851197 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.851212 (+ 15us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.851253 (+ 41us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.851308 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":99,"threads_started":1}
I20240501 19:56:40.851485 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.847280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:41839 (local address 127.0.0.1:36814)
0501 19:56:40.847733 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.847746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.847827 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.848491 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.848494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.848524 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:40.848792 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.848801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.850239 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.850243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.851162 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.851172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.851270 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.851287 (+ 17us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.851333 (+ 46us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.851376 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"spinlock_wait_cycles":3200,"thread_start_us":99,"threads_started":1}
I20240501 19:56:40.852480 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.847795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36226 (local address 127.8.168.65:46591)
0501 19:56:40.848475 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.848481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.848499 (+ 18us) server_negotiation.cc:407] Connection header received
0501 19:56:40.848575 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.848579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.848764 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.848913 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:40.849401 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.850398 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.851612 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.852125 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.852236 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.852298 (+ 62us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.852366 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":109,"threads_started":1}
I20240501 19:56:40.852533 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.847683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36814 (local address 127.8.168.66:41839)
0501 19:56:40.847987 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.847994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.848018 (+ 24us) server_negotiation.cc:407] Connection header received
0501 19:56:40.848115 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.848120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.848321 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.848918 (+ 597us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:40.849041 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.850128 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.851795 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.852219 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.852304 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.852359 (+ 55us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.852422 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
I20240501 19:56:40.853490 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.847489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57916 (local address 127.8.168.67:37273)
0501 19:56:40.848052 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.848058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.848080 (+ 22us) server_negotiation.cc:407] Connection header received
0501 19:56:40.848167 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.848173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.848339 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.848483 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:40.848885 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.849527 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.852756 (+ 3229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.853134 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.853212 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.853268 (+ 56us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.853357 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":104,"threads_started":1}
I20240501 19:56:40.855216 16040 tablet_service.cc:1467] Processing CreateTablet for tablet 07d8b01f5b13419f9f1a4586ce38ea9c (DEFAULT_TABLE table=test-table [id=a54c2979905d4ab9ac4c27bcbeed10f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.855588 16176 tablet_service.cc:1467] Processing CreateTablet for tablet 07d8b01f5b13419f9f1a4586ce38ea9c (DEFAULT_TABLE table=test-table [id=a54c2979905d4ab9ac4c27bcbeed10f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.856088 16040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d8b01f5b13419f9f1a4586ce38ea9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.856412 16176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d8b01f5b13419f9f1a4586ce38ea9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.858460 15903 tablet_service.cc:1467] Processing CreateTablet for tablet 07d8b01f5b13419f9f1a4586ce38ea9c (DEFAULT_TABLE table=test-table [id=a54c2979905d4ab9ac4c27bcbeed10f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:40.859361 15903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d8b01f5b13419f9f1a4586ce38ea9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.862329 16272 tablet_bootstrap.cc:492] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f: Bootstrap starting.
I20240501 19:56:40.863927 16273 tablet_bootstrap.cc:492] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c: Bootstrap starting.
I20240501 19:56:40.864882 16272 tablet_bootstrap.cc:654] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.865537 16272 log.cc:824] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:40.865923 16273 tablet_bootstrap.cc:654] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.866150 16274 tablet_bootstrap.cc:492] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f: Bootstrap starting.
I20240501 19:56:40.866835 16273 log.cc:824] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:40.867403 16272 tablet_bootstrap.cc:492] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f: No bootstrap required, opened a new log
I20240501 19:56:40.867590 16272 ts_tablet_manager.cc:1367] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 19:56:40.868463 16273 tablet_bootstrap.cc:492] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c: No bootstrap required, opened a new log
I20240501 19:56:40.868628 16273 ts_tablet_manager.cc:1367] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 19:56:40.868806 16274 tablet_bootstrap.cc:654] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.869580 16274 log.cc:824] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:40.870689 16272 raft_consensus.cc:348] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } }
I20240501 19:56:40.870903 16272 raft_consensus.cc:374] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.870960 16272 raft_consensus.cc:729] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e74f065d34341d8a34c0b20761a558f, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.871232 16273 raft_consensus.cc:348] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } }
I20240501 19:56:40.871390 16273 raft_consensus.cc:374] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.871436 16274 tablet_bootstrap.cc:492] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f: No bootstrap required, opened a new log
I20240501 19:56:40.871438 16273 raft_consensus.cc:729] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2522fdb3bfeb457e9a395935914b896c, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.871400 16272 consensus_queue.cc:260] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } }
I20240501 19:56:40.871594 16274 ts_tablet_manager.cc:1367] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240501 19:56:40.871815 16273 consensus_queue.cc:260] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } }
I20240501 19:56:40.872152 16107 heartbeater.cc:503] Master 127.8.168.126:45289 was elected leader, sending a full tablet report...
I20240501 19:56:40.872586 16273 ts_tablet_manager.cc:1398] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 19:56:40.873176 16244 heartbeater.cc:503] Master 127.8.168.126:45289 was elected leader, sending a full tablet report...
I20240501 19:56:40.874331 16272 ts_tablet_manager.cc:1398] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
W20240501 19:56:40.877507 16109 tablet.cc:2369] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:40.878221 16274 raft_consensus.cc:348] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } }
I20240501 19:56:40.878438 16274 raft_consensus.cc:374] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.878492 16274 raft_consensus.cc:729] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3247c2f74cd44a12a587f056979b798f, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.878948 16274 consensus_queue.cc:260] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } }
I20240501 19:56:40.879565 15970 heartbeater.cc:503] Master 127.8.168.126:45289 was elected leader, sending a full tablet report...
I20240501 19:56:40.879813 16274 ts_tablet_manager.cc:1398] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20240501 19:56:40.917284 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.914852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:40517 (local address 127.0.0.1:37512)
0501 19:56:40.914974 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.914989 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.915069 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.915338 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.915340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.915357 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:40.915536 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.915548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.916274 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.916276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.916944 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.916954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.917055 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.917077 (+ 22us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.917117 (+ 40us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.917170 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20240501 19:56:40.917781 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.914958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.8.168.124:40517)
0501 19:56:40.915082 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.915086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.915102 (+ 16us) server_negotiation.cc:407] Connection header received
0501 19:56:40.915153 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.915156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.915221 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.915321 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:40.915632 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.916186 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.917082 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.917455 (+ 373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.917587 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.917633 (+ 46us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.917684 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240501 19:56:40.918624 15803 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20240501 19:56:40.921180 16281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fd6981a1f214470ab279f88136f1b14. Latest consensus state: current_term: 1 leader_uuid: "9fd6981a1f214470ab279f88136f1b14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: NON_VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: true } } }
I20240501 19:56:40.921308 16281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [sys.catalog]: This master's current role is: LEARNER
W20240501 19:56:40.938041 15972 tablet.cc:2369] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:40.948307 16279 raft_consensus.cc:482] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:40.948508 16279 raft_consensus.cc:504] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } }
I20240501 19:56:40.949663 16279 leader_election.cc:288] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e74f065d34341d8a34c0b20761a558f (127.8.168.66:41839), 3247c2f74cd44a12a587f056979b798f (127.8.168.65:46591)
I20240501 19:56:40.952581 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.949902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:46591 (local address 127.8.168.67:59631)
0501 19:56:40.950074 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.950087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.950163 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.950508 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.950511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.950533 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:40.950783 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.950790 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.951583 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.951586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.952210 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.952218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.952322 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.952344 (+ 22us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.952392 (+ 48us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.952445 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20240501 19:56:40.952580 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.949848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:41839 (local address 127.8.168.67:52203)
0501 19:56:40.950017 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:40.950031 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:40.950124 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:40.950712 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:40.950715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:40.950730 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:40.950954 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.950962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.951668 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.951671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:40.952206 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:40.952212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.952328 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.952345 (+ 17us) client_negotiation.cc:769] Sending connection context
0501 19:56:40.952388 (+ 43us) client_negotiation.cc:240] Negotiation successful
0501 19:56:40.952452 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20240501 19:56:40.953100 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.950086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:59631 (local address 127.8.168.65:46591)
0501 19:56:40.950203 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.950208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.950221 (+ 13us) server_negotiation.cc:407] Connection header received
0501 19:56:40.950268 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.950270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.950317 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.950396 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:40.950900 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.951472 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.952334 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.952859 (+ 525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.952899 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.952947 (+ 48us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.952996 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20240501 19:56:40.953326 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.950335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:52203 (local address 127.8.168.66:41839)
0501 19:56:40.950490 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.950494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.950506 (+ 12us) server_negotiation.cc:407] Connection header received
0501 19:56:40.950559 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.950562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.950607 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.950686 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:40.951072 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.951565 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.952479 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.953089 (+ 610us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.953120 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.953163 (+ 43us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.953215 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20240501 19:56:40.953668 15923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d8b01f5b13419f9f1a4586ce38ea9c" candidate_uuid: "2522fdb3bfeb457e9a395935914b896c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3247c2f74cd44a12a587f056979b798f" is_pre_election: true
I20240501 19:56:40.953859 16060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d8b01f5b13419f9f1a4586ce38ea9c" candidate_uuid: "2522fdb3bfeb457e9a395935914b896c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e74f065d34341d8a34c0b20761a558f" is_pre_election: true
I20240501 19:56:40.953975 15923 raft_consensus.cc:2454] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2522fdb3bfeb457e9a395935914b896c in term 0.
I20240501 19:56:40.954111 16060 raft_consensus.cc:2454] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2522fdb3bfeb457e9a395935914b896c in term 0.
I20240501 19:56:40.954420 16129 leader_election.cc:302] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2522fdb3bfeb457e9a395935914b896c, 3247c2f74cd44a12a587f056979b798f; no voters:
I20240501 19:56:40.954654 16279 raft_consensus.cc:2781] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:40.954721 16279 raft_consensus.cc:482] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:40.954766 16279 raft_consensus.cc:3037] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.956036 16279 raft_consensus.cc:504] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } }
I20240501 19:56:40.956405 16279 leader_election.cc:288] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [CANDIDATE]: Term 1 election: Requested vote from peers 2e74f065d34341d8a34c0b20761a558f (127.8.168.66:41839), 3247c2f74cd44a12a587f056979b798f (127.8.168.65:46591)
I20240501 19:56:40.956779 16060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d8b01f5b13419f9f1a4586ce38ea9c" candidate_uuid: "2522fdb3bfeb457e9a395935914b896c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e74f065d34341d8a34c0b20761a558f"
I20240501 19:56:40.956825 15923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d8b01f5b13419f9f1a4586ce38ea9c" candidate_uuid: "2522fdb3bfeb457e9a395935914b896c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3247c2f74cd44a12a587f056979b798f"
I20240501 19:56:40.956931 15923 raft_consensus.cc:3037] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.956931 16060 raft_consensus.cc:3037] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.958127 15923 raft_consensus.cc:2454] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2522fdb3bfeb457e9a395935914b896c in term 1.
I20240501 19:56:40.958452 16129 leader_election.cc:302] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2522fdb3bfeb457e9a395935914b896c, 3247c2f74cd44a12a587f056979b798f; no voters:
I20240501 19:56:40.958487 16060 raft_consensus.cc:2454] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2522fdb3bfeb457e9a395935914b896c in term 1.
I20240501 19:56:40.958655 16279 raft_consensus.cc:2781] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:40.958864 16279 raft_consensus.cc:686] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [term 1 LEADER]: Becoming Leader. State: Replica: 2522fdb3bfeb457e9a395935914b896c, State: Running, Role: LEADER
I20240501 19:56:40.959187 16279 consensus_queue.cc:237] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } }
I20240501 19:56:40.962570 15000 catalog_manager.cc:5484] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2522fdb3bfeb457e9a395935914b896c (127.8.168.67). New cstate: current_term: 1 leader_uuid: "2522fdb3bfeb457e9a395935914b896c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:40.964236 16261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fd6981a1f214470ab279f88136f1b14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: NON_VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: true } } }
I20240501 19:56:40.964360 16261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:40.965827 16261 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14: attempting to promote NON_VOTER 697109cc171e40c1bb4afb53014ca1bc to VOTER
I20240501 19:56:40.966324 16261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false } }
I20240501 19:56:40.969254 15803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 1 LEARNER]: Refusing update from remote peer 9fd6981a1f214470ab279f88136f1b14: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:40.969292 15611 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 1 FOLLOWER]: Refusing update from remote peer 9fd6981a1f214470ab279f88136f1b14: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:40.969677 16261 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:40.969828 16290 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:40.971264 16290 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 697109cc171e40c1bb4afb53014ca1bc (127.8.168.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false } } }
I20240501 19:56:40.971796 15611 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 697109cc171e40c1bb4afb53014ca1bc (127.8.168.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false } } }
I20240501 19:56:40.972149 15803 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 697109cc171e40c1bb4afb53014ca1bc (127.8.168.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false } } }
I20240501 19:56:40.972980 16265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9fd6981a1f214470ab279f88136f1b14" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false } } }
I20240501 19:56:40.973068 16265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:40.973255 16265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fd6981a1f214470ab279f88136f1b14" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false } } }
I20240501 19:56:40.973325 16265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:40.973475 16265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fd6981a1f214470ab279f88136f1b14" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false } } }
I20240501 19:56:40.973551 16265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:40.974134 16281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fd6981a1f214470ab279f88136f1b14" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false } } }
I20240501 19:56:40.974248 16281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:56:40.980945 16291 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fd6981a1f214470ab279f88136f1b14" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false } } }
I20240501 19:56:40.981099 16291 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:56:40.982086 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:40.978584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57924 (local address 127.8.168.67:37273)
0501 19:56:40.978696 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:40.978700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:40.978731 (+ 31us) server_negotiation.cc:407] Connection header received
0501 19:56:40.978903 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:40.978905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:40.979012 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:40.979114 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:40.979489 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.980037 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:40.980951 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:40.981148 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:40.981257 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:40.981681 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:40.981778 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:40.981873 (+ 95us) server_negotiation.cc:299] Negotiation successful
0501 19:56:40.981920 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240501 19:56:40.987751 16060 raft_consensus.cc:1261] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f [term 1 FOLLOWER]: Refusing update from remote peer 2522fdb3bfeb457e9a395935914b896c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:40.987767 15923 raft_consensus.cc:1261] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f [term 1 FOLLOWER]: Refusing update from remote peer 2522fdb3bfeb457e9a395935914b896c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:40.988355 16286 consensus_queue.cc:1035] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:40.988521 16279 consensus_queue.cc:1035] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:40.999974 16295 mvcc.cc:205] Tried to move back new op lower bound from 7022974570438643712 to 7022974570331488256. Current Snapshot: MvccSnapshot[applied={T|T < 7022974570438643712 or (T in {7022974570438643712})}]
I20240501 19:56:41.006323 14999 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33440:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:41.006500 14999 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33440:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:41.010922 14999 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 9fd6981a1f214470ab279f88136f1b14: Sending DeleteTablet for 3 replicas of tablet 07d8b01f5b13419f9f1a4586ce38ea9c
I20240501 19:56:41.012064 16040 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d8b01f5b13419f9f1a4586ce38ea9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:41 UTC) from {username='oryx', principal='oryx/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:36814
I20240501 19:56:41.012566 16302 tablet_replica.cc:330] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f: stopping tablet replica
I20240501 19:56:41.012751 15903 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d8b01f5b13419f9f1a4586ce38ea9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:41 UTC) from {username='oryx', principal='oryx/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:36226
I20240501 19:56:41.013120 16302 raft_consensus.cc:2229] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:41.013538 16302 raft_consensus.cc:2258] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:41.013702 16176 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d8b01f5b13419f9f1a4586ce38ea9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:41 UTC) from {username='oryx', principal='oryx/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:57916
I20240501 19:56:41.015132 16302 ts_tablet_manager.cc:1875] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:41.016911 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 14969
I20240501 19:56:41.017957 16302 ts_tablet_manager.cc:1888] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:41.018119 16302 log.cc:1196] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/07d8b01f5b13419f9f1a4586ce38ea9c
I20240501 19:56:41.018523 16302 ts_tablet_manager.cc:1909] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2e74f065d34341d8a34c0b20761a558f: Deleting consensus metadata
I20240501 19:56:41.019877 16303 tablet_replica.cc:330] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f: stopping tablet replica
I20240501 19:56:41.020129 16303 raft_consensus.cc:2229] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:41.020421 16303 raft_consensus.cc:2258] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:41.020783 16304 tablet_replica.cc:330] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c: stopping tablet replica
I20240501 19:56:41.020995 16304 raft_consensus.cc:2229] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:41.021291 16304 raft_consensus.cc:2258] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:41.022687 16303 ts_tablet_manager.cc:1875] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 19:56:41.025626 15774 connection.cc:535] server connection from 127.0.0.1:37512 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 19:56:41.026146 16303 ts_tablet_manager.cc:1888] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:41.026242 16303 log.cc:1196] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-0/wal/wals/07d8b01f5b13419f9f1a4586ce38ea9c
I20240501 19:56:41.026860 16303 ts_tablet_manager.cc:1909] T 07d8b01f5b13419f9f1a4586ce38ea9c P 3247c2f74cd44a12a587f056979b798f: Deleting consensus metadata
I20240501 19:56:41.028851 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:41.015730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43172 (local address 127.8.168.125:36409)
0501 19:56:41.015866 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:41.015870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:41.018361 (+ 2491us) server_negotiation.cc:407] Connection header received
0501 19:56:41.018613 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:41.018617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:41.018671 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:41.018727 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:41.021698 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:41.022413 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:41.023324 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:41.023521 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:41.025437 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:41.025457 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:41.025460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:41.025490 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:41.027227 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:41.027700 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:41.027703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:41.027705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:41.027755 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:41.028008 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:41.028012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:41.028013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:41.028178 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:41.028265 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:41.028554 (+ 289us) server_negotiation.cc:299] Negotiation successful
0501 19:56:41.028668 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20240501 19:56:41.032861 15859 connection.cc:535] server connection from 127.0.0.1:36226 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:41.033252 15993 connection.cc:535] server connection from 127.0.0.1:36814 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 19:56:41.033844 16304 ts_tablet_manager.cc:1875] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:41.036314 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:41.017952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.8.168.124:40517)
0501 19:56:41.018200 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:41.018203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:41.029201 (+ 10998us) server_negotiation.cc:407] Connection header received
0501 19:56:41.029372 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:41.029376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:41.029427 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:41.029477 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:41.030293 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:41.030784 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:41.031359 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:41.031830 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:41.032447 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:41.032464 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:41.032466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:41.032489 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:41.034381 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:41.035053 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:41.035057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:41.035059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:41.035112 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:41.035372 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:41.035376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:41.035378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:41.035554 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:41.035658 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:41.036001 (+ 343us) server_negotiation.cc:299] Negotiation successful
0501 19:56:41.036152 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20240501 19:56:41.036586 16304 ts_tablet_manager.cc:1888] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:41.036682 16304 log.cc:1196] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/07d8b01f5b13419f9f1a4586ce38ea9c
I20240501 19:56:41.037015 16304 ts_tablet_manager.cc:1909] T 07d8b01f5b13419f9f1a4586ce38ea9c P 2522fdb3bfeb457e9a395935914b896c: Deleting consensus metadata
W20240501 19:56:41.038774 16131 connection.cc:441] server connection from 127.0.0.1:57916 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57916 (request call id 1) could send its response
I20240501 19:56:41.045820 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:41.037190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.8.168.125:36409)
0501 19:56:41.037318 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:41.037321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:41.037832 (+ 511us) server_negotiation.cc:407] Connection header received
0501 19:56:41.038001 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:41.038005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:41.038066 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:41.038116 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:41.039593 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:41.040283 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:41.041069 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:41.041264 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:41.042729 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:41.042748 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:41.042750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:41.042777 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:41.044431 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:41.044853 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:41.044856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:41.044858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:41.044901 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:41.045104 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:41.045107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:41.045109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:41.045241 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:41.045356 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:41.045553 (+ 197us) server_negotiation.cc:299] Negotiation successful
0501 19:56:41.045655 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20240501 19:56:41.046566 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:41.037870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.8.168.124:40517)
0501 19:56:41.038009 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:41.038012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:41.038971 (+ 959us) server_negotiation.cc:407] Connection header received
0501 19:56:41.039115 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:41.039118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:41.039161 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:41.039226 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:41.040164 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:41.040938 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:41.041785 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:41.042346 (+ 561us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:41.043124 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:41.043145 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:41.043149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:41.043177 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:41.044968 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:41.045452 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:41.045455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:41.045457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:41.045503 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:41.045729 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:41.045732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:41.045734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:41.045888 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:41.045977 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:41.046311 (+ 334us) server_negotiation.cc:299] Negotiation successful
0501 19:56:41.046423 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20240501 19:56:42.001526 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.001056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.65:48585)
0501 19:56:42.001321 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.001395 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
W20240501 19:56:42.001847 15970 heartbeater.cc:644] Failed to heartbeat to 127.8.168.126:45289 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
I20240501 19:56:42.003883 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.003399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.67:37899)
0501 19:56:42.003689 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.003766 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":68,"threads_started":1}
W20240501 19:56:42.004120 16244 heartbeater.cc:644] Failed to heartbeat to 127.8.168.126:45289 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
I20240501 19:56:42.004298 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.003870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.66:38027)
0501 19:56:42.004142 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.004213 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":57,"threads_started":1}
W20240501 19:56:42.004551 16107 heartbeater.cc:644] Failed to heartbeat to 127.8.168.126:45289 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
I20240501 19:56:42.511860 16316 raft_consensus.cc:482] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9fd6981a1f214470ab279f88136f1b14)
I20240501 19:56:42.512051 16316 raft_consensus.cc:504] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false } }
I20240501 19:56:42.513082 16316 leader_election.cc:288] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9fd6981a1f214470ab279f88136f1b14 (127.8.168.126:45289), 697109cc171e40c1bb4afb53014ca1bc (127.8.168.124:40517)
I20240501 19:56:42.513748 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.513259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.0.0.1:33888)
0501 19:56:42.513560 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.513640 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":74,"threads_started":1}
W20240501 19:56:42.513935 15589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
I20240501 19:56:42.514689 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.514463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.0.0.1:33896)
0501 19:56:42.514574 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.514623 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20240501 19:56:42.514807 15589 leader_election.cc:334] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9fd6981a1f214470ab279f88136f1b14 (127.8.168.126:45289): Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
I20240501 19:56:42.515985 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.513259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:40517 (local address 127.0.0.1:37532)
0501 19:56:42.513582 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.513594 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:42.513682 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:42.513930 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:42.513933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:42.513951 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:42.514169 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:42.514176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.515022 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:42.515026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:42.515622 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:42.515630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.515733 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:42.515755 (+ 22us) client_negotiation.cc:769] Sending connection context
0501 19:56:42.515804 (+ 49us) client_negotiation.cc:240] Negotiation successful
0501 19:56:42.515864 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":43,"thread_start_us":59,"threads_started":1}
I20240501 19:56:42.516427 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.513477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.8.168.124:40517)
0501 19:56:42.513690 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:42.513694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:42.513711 (+ 17us) server_negotiation.cc:407] Connection header received
0501 19:56:42.513764 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:42.513767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:42.513822 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:42.513921 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:42.514286 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.514921 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:42.515738 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.516201 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:42.516236 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:42.516280 (+ 44us) server_negotiation.cc:299] Negotiation successful
0501 19:56:42.516327 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":69,"threads_started":1}
I20240501 19:56:42.516870 15803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "697109cc171e40c1bb4afb53014ca1bc" is_pre_election: true
I20240501 19:56:42.517351 15590 leader_election.cc:302] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a88d3cb5c6774a43acf84bda9c2362e1; no voters: 697109cc171e40c1bb4afb53014ca1bc, 9fd6981a1f214470ab279f88136f1b14
I20240501 19:56:42.517570 16316 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:42.523048 16321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9fd6981a1f214470ab279f88136f1b14)
I20240501 19:56:42.523214 16321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false } }
I20240501 19:56:42.524233 16321 leader_election.cc:288] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9fd6981a1f214470ab279f88136f1b14 (127.8.168.126:45289), a88d3cb5c6774a43acf84bda9c2362e1 (127.8.168.125:36409)
I20240501 19:56:42.524794 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.524374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.0.0.1:33908)
0501 19:56:42.524599 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.524677 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":64,"threads_started":1}
W20240501 19:56:42.524926 15774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
I20240501 19:56:42.525714 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.525496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.0.0.1:33914)
0501 19:56:42.525600 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.525647 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20240501 19:56:42.525851 15774 leader_election.cc:334] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9fd6981a1f214470ab279f88136f1b14 (127.8.168.126:45289): Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
I20240501 19:56:42.527088 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.524374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36409 (local address 127.0.0.1:43186)
0501 19:56:42.524666 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.524679 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:42.524814 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:42.525077 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:42.525080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:42.525099 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:42.525290 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:42.525297 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.526128 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:42.526131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:42.526701 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:42.526708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.526811 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:42.526831 (+ 20us) client_negotiation.cc:769] Sending connection context
0501 19:56:42.526879 (+ 48us) client_negotiation.cc:240] Negotiation successful
0501 19:56:42.526947 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":34,"thread_start_us":49,"threads_started":1}
I20240501 19:56:42.527480 16324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.524500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.8.168.125:36409)
0501 19:56:42.524824 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:42.524827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:42.524842 (+ 15us) server_negotiation.cc:407] Connection header received
0501 19:56:42.524918 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:42.524920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:42.524960 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:42.525029 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:42.525397 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.526024 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:42.526813 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.527270 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:42.527304 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:42.527347 (+ 43us) server_negotiation.cc:299] Negotiation successful
0501 19:56:42.527388 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":114,"threads_started":1}
I20240501 19:56:42.527884 15611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "697109cc171e40c1bb4afb53014ca1bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" is_pre_election: true
I20240501 19:56:42.528057 15611 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 697109cc171e40c1bb4afb53014ca1bc in term 1.
I20240501 19:56:42.528388 15775 leader_election.cc:302] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 697109cc171e40c1bb4afb53014ca1bc, a88d3cb5c6774a43acf84bda9c2362e1; no voters: 9fd6981a1f214470ab279f88136f1b14
I20240501 19:56:42.528590 16321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:56:42.528659 16321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9fd6981a1f214470ab279f88136f1b14)
I20240501 19:56:42.528698 16321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:42.529666 16321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false } }
I20240501 19:56:42.529992 16321 leader_election.cc:288] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [CANDIDATE]: Term 2 election: Requested vote from peers 9fd6981a1f214470ab279f88136f1b14 (127.8.168.126:45289), a88d3cb5c6774a43acf84bda9c2362e1 (127.8.168.125:36409)
I20240501 19:56:42.530345 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.530118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.0.0.1:33928)
0501 19:56:42.530241 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.530293 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20240501 19:56:42.530347 15611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "697109cc171e40c1bb4afb53014ca1bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a88d3cb5c6774a43acf84bda9c2362e1"
I20240501 19:56:42.530445 15611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:42.530927 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.530762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.0.0.1:33940)
0501 19:56:42.530844 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.530881 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20240501 19:56:42.531075 15774 leader_election.cc:334] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9fd6981a1f214470ab279f88136f1b14 (127.8.168.126:45289): Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
I20240501 19:56:42.531435 15611 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 697109cc171e40c1bb4afb53014ca1bc in term 2.
I20240501 19:56:42.531673 15775 leader_election.cc:302] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 697109cc171e40c1bb4afb53014ca1bc, a88d3cb5c6774a43acf84bda9c2362e1; no voters: 9fd6981a1f214470ab279f88136f1b14
I20240501 19:56:42.531803 16321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:42.531985 16321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [term 2 LEADER]: Becoming Leader. State: Replica: 697109cc171e40c1bb4afb53014ca1bc, State: Running, Role: LEADER
I20240501 19:56:42.532199 16321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false } }
I20240501 19:56:42.533186 16326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 697109cc171e40c1bb4afb53014ca1bc. Latest consensus state: current_term: 2 leader_uuid: "697109cc171e40c1bb4afb53014ca1bc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false } } }
I20240501 19:56:42.533295 16326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:42.533577 16328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:42.535529 16328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:42.535925 16328 catalog_manager.cc:1259] Loaded cluster ID: cb2d94b3f8bd46e9bbe19926806e1c00
I20240501 19:56:42.535971 16328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:42.536490 16328 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:42.536886 16328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: Loaded TSK: 0
I20240501 19:56:42.537544 16328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:56:42.559191 15793 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:56:42.561772 15793 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:42.564574 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.564334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.0.0.1:33956)
0501 19:56:42.564440 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.564507 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20240501 19:56:42.564651 15611 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [term 2 FOLLOWER]: Refusing update from remote peer 697109cc171e40c1bb4afb53014ca1bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240501 19:56:42.565274 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.565081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.0.0.1:33960)
0501 19:56:42.565176 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.565219 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20240501 19:56:42.565385 15774 consensus_peers.cc:485] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc -> Peer 9fd6981a1f214470ab279f88136f1b14 (127.8.168.126:45289): Couldn't send request to peer 9fd6981a1f214470ab279f88136f1b14. Status: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:42.565404 16326 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:42.566802 16316 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 697109cc171e40c1bb4afb53014ca1bc. Latest consensus state: current_term: 2 leader_uuid: "697109cc171e40c1bb4afb53014ca1bc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false } } }
I20240501 19:56:42.566908 16316 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:56:42.567909 16321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "697109cc171e40c1bb4afb53014ca1bc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false } } }
I20240501 19:56:42.568056 16321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:42.568483 16316 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "697109cc171e40c1bb4afb53014ca1bc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9fd6981a1f214470ab279f88136f1b14" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45289 } } peers { permanent_uuid: "a88d3cb5c6774a43acf84bda9c2362e1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36409 } attrs { promote: false } } peers { permanent_uuid: "697109cc171e40c1bb4afb53014ca1bc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 40517 } attrs { promote: false } } }
I20240501 19:56:42.568584 16316 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88d3cb5c6774a43acf84bda9c2362e1 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:56:42.576141 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.573210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:46591 (local address 127.0.0.1:36228)
0501 19:56:42.573318 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.573330 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:42.573393 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:42.573873 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:42.573876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:42.573894 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:42.574133 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:42.574142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.575185 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:42.575189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:42.575797 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:42.575804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.575899 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:42.575919 (+ 20us) client_negotiation.cc:769] Sending connection context
0501 19:56:42.575960 (+ 41us) client_negotiation.cc:240] Negotiation successful
0501 19:56:42.576014 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20240501 19:56:42.576987 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.573704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:37273 (local address 127.0.0.1:57926)
0501 19:56:42.573925 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.573939 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:42.574021 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:42.574551 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:42.574555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:42.574572 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:42.574804 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:42.574812 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.575783 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:42.575786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:42.576614 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:42.576623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.576756 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:42.576773 (+ 17us) client_negotiation.cc:769] Sending connection context
0501 19:56:42.576821 (+ 48us) client_negotiation.cc:240] Negotiation successful
0501 19:56:42.576879 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20240501 19:56:42.577206 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.573354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36228 (local address 127.8.168.65:46591)
0501 19:56:42.573610 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:42.573613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:42.573643 (+ 30us) server_negotiation.cc:407] Connection header received
0501 19:56:42.573693 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:42.573696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:42.573747 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:42.574239 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:42.574349 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.575043 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:42.576439 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.576937 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:42.576972 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:42.577024 (+ 52us) server_negotiation.cc:299] Negotiation successful
0501 19:56:42.577085 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
I20240501 19:56:42.577730 15903 tablet_service.cc:1467] Processing CreateTablet for tablet 0605625dbbe149538a96fde4aaf1d3ce (DEFAULT_TABLE table=test-table [id=6e3cd111ea7f456c81e72302733d7157]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.577872 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.574086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.8.168.67:37273)
0501 19:56:42.574303 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:42.574307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:42.574322 (+ 15us) server_negotiation.cc:407] Connection header received
0501 19:56:42.574371 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:42.574373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:42.574422 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:42.574898 (+ 476us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:42.574986 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.575675 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:42.577306 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.577660 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:42.577689 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:42.577731 (+ 42us) server_negotiation.cc:299] Negotiation successful
0501 19:56:42.577775 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":63,"threads_started":1}
I20240501 19:56:42.577988 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.573873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36830 (local address 127.8.168.66:41839)
0501 19:56:42.574513 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:42.574517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:42.574531 (+ 14us) server_negotiation.cc:407] Connection header received
0501 19:56:42.574629 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:42.574632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:42.574677 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:42.574753 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:42.575142 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.575856 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:42.577040 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.577720 (+ 680us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:42.577757 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:42.577807 (+ 50us) server_negotiation.cc:299] Negotiation successful
0501 19:56:42.577868 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":81,"threads_started":1}
I20240501 19:56:42.578035 15903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0605625dbbe149538a96fde4aaf1d3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.578065 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.573704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:41839 (local address 127.0.0.1:36830)
0501 19:56:42.574451 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.574464 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:42.574532 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:42.574771 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:42.574774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:42.574789 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:42.575025 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:42.575032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.575950 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:42.575954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:42.576656 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:42.576665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.576800 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:42.576819 (+ 19us) client_negotiation.cc:769] Sending connection context
0501 19:56:42.576868 (+ 49us) client_negotiation.cc:240] Negotiation successful
0501 19:56:42.576931 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":502,"thread_start_us":85,"threads_started":1}
I20240501 19:56:42.578351 16176 tablet_service.cc:1467] Processing CreateTablet for tablet 0605625dbbe149538a96fde4aaf1d3ce (DEFAULT_TABLE table=test-table [id=6e3cd111ea7f456c81e72302733d7157]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.578593 16176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0605625dbbe149538a96fde4aaf1d3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.580416 16338 tablet_bootstrap.cc:492] T 0605625dbbe149538a96fde4aaf1d3ce P 3247c2f74cd44a12a587f056979b798f: Bootstrap starting.
I20240501 19:56:42.581331 16338 tablet_bootstrap.cc:654] T 0605625dbbe149538a96fde4aaf1d3ce P 3247c2f74cd44a12a587f056979b798f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.581286 16040 tablet_service.cc:1467] Processing CreateTablet for tablet 0605625dbbe149538a96fde4aaf1d3ce (DEFAULT_TABLE table=test-table [id=6e3cd111ea7f456c81e72302733d7157]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.581344 16339 tablet_bootstrap.cc:492] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c: Bootstrap starting.
I20240501 19:56:42.581561 16040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0605625dbbe149538a96fde4aaf1d3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.582110 16339 tablet_bootstrap.cc:654] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.582510 16338 tablet_bootstrap.cc:492] T 0605625dbbe149538a96fde4aaf1d3ce P 3247c2f74cd44a12a587f056979b798f: No bootstrap required, opened a new log
I20240501 19:56:42.582605 16338 ts_tablet_manager.cc:1367] T 0605625dbbe149538a96fde4aaf1d3ce P 3247c2f74cd44a12a587f056979b798f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:42.583009 16338 raft_consensus.cc:348] T 0605625dbbe149538a96fde4aaf1d3ce P 3247c2f74cd44a12a587f056979b798f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } }
I20240501 19:56:42.583094 16339 tablet_bootstrap.cc:492] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c: No bootstrap required, opened a new log
I20240501 19:56:42.583133 16338 raft_consensus.cc:374] T 0605625dbbe149538a96fde4aaf1d3ce P 3247c2f74cd44a12a587f056979b798f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.583164 16339 ts_tablet_manager.cc:1367] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:42.583168 16338 raft_consensus.cc:729] T 0605625dbbe149538a96fde4aaf1d3ce P 3247c2f74cd44a12a587f056979b798f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3247c2f74cd44a12a587f056979b798f, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.583325 16338 consensus_queue.cc:260] T 0605625dbbe149538a96fde4aaf1d3ce P 3247c2f74cd44a12a587f056979b798f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } }
I20240501 19:56:42.583598 16339 raft_consensus.cc:348] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } }
I20240501 19:56:42.583846 16339 raft_consensus.cc:374] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.583891 16339 raft_consensus.cc:729] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2522fdb3bfeb457e9a395935914b896c, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.583999 16338 ts_tablet_manager.cc:1398] T 0605625dbbe149538a96fde4aaf1d3ce P 3247c2f74cd44a12a587f056979b798f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:42.584033 16339 consensus_queue.cc:260] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } }
I20240501 19:56:42.584317 16339 ts_tablet_manager.cc:1398] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.584748 16342 tablet_bootstrap.cc:492] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f: Bootstrap starting.
I20240501 19:56:42.585623 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.585223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.67:47263)
0501 19:56:42.585483 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.585535 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20240501 19:56:42.586676 15969 heartbeater.cc:503] Master 127.8.168.124:40517 was elected leader, sending a full tablet report...
I20240501 19:56:42.587005 16342 tablet_bootstrap.cc:654] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.587440 16243 heartbeater.cc:503] Master 127.8.168.124:40517 was elected leader, sending a full tablet report...
I20240501 19:56:42.587529 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.584392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.65:39197)
0501 19:56:42.585415 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.585471 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":942,"thread_start_us":87,"threads_started":1}
I20240501 19:56:42.588215 16342 tablet_bootstrap.cc:492] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f: No bootstrap required, opened a new log
I20240501 19:56:42.588299 16342 ts_tablet_manager.cc:1367] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:56:42.588656 16342 raft_consensus.cc:348] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } }
I20240501 19:56:42.588769 16342 raft_consensus.cc:374] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.588809 16342 raft_consensus.cc:729] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e74f065d34341d8a34c0b20761a558f, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.588935 16342 consensus_queue.cc:260] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } }
I20240501 19:56:42.589241 16342 ts_tablet_manager.cc:1398] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.589517 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.587953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.65:35305)
0501 19:56:42.589400 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.589449 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1376}
I20240501 19:56:42.589620 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.589373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.67:47469)
0501 19:56:42.589507 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.589553 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20240501 19:56:42.590513 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.590145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.66:33483)
0501 19:56:42.590371 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.590422 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
I20240501 19:56:42.591042 16106 heartbeater.cc:503] Master 127.8.168.124:40517 was elected leader, sending a full tablet report...
I20240501 19:56:42.591351 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.590992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.66:60085)
0501 19:56:42.591242 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.591286 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
I20240501 19:56:42.773772 16344 raft_consensus.cc:482] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:42.773918 16344 raft_consensus.cc:504] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } }
I20240501 19:56:42.774359 16344 leader_election.cc:288] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3247c2f74cd44a12a587f056979b798f (127.8.168.65:46591), 2e74f065d34341d8a34c0b20761a558f (127.8.168.66:41839)
I20240501 19:56:42.774852 16060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0605625dbbe149538a96fde4aaf1d3ce" candidate_uuid: "2522fdb3bfeb457e9a395935914b896c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e74f065d34341d8a34c0b20761a558f" is_pre_election: true
I20240501 19:56:42.774883 15923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0605625dbbe149538a96fde4aaf1d3ce" candidate_uuid: "2522fdb3bfeb457e9a395935914b896c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3247c2f74cd44a12a587f056979b798f" is_pre_election: true
I20240501 19:56:42.774982 16060 raft_consensus.cc:2454] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2522fdb3bfeb457e9a395935914b896c in term 0.
I20240501 19:56:42.775034 15923 raft_consensus.cc:2454] T 0605625dbbe149538a96fde4aaf1d3ce P 3247c2f74cd44a12a587f056979b798f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2522fdb3bfeb457e9a395935914b896c in term 0.
I20240501 19:56:42.775280 16132 leader_election.cc:302] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2522fdb3bfeb457e9a395935914b896c, 2e74f065d34341d8a34c0b20761a558f; no voters:
I20240501 19:56:42.775470 16344 raft_consensus.cc:2781] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:42.775533 16344 raft_consensus.cc:482] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:42.775578 16344 raft_consensus.cc:3037] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.776455 16344 raft_consensus.cc:504] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } }
I20240501 19:56:42.776820 16344 leader_election.cc:288] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [CANDIDATE]: Term 1 election: Requested vote from peers 3247c2f74cd44a12a587f056979b798f (127.8.168.65:46591), 2e74f065d34341d8a34c0b20761a558f (127.8.168.66:41839)
I20240501 19:56:42.777160 16060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0605625dbbe149538a96fde4aaf1d3ce" candidate_uuid: "2522fdb3bfeb457e9a395935914b896c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e74f065d34341d8a34c0b20761a558f"
I20240501 19:56:42.777163 15923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0605625dbbe149538a96fde4aaf1d3ce" candidate_uuid: "2522fdb3bfeb457e9a395935914b896c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3247c2f74cd44a12a587f056979b798f"
I20240501 19:56:42.777278 16060 raft_consensus.cc:3037] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.777278 15923 raft_consensus.cc:3037] T 0605625dbbe149538a96fde4aaf1d3ce P 3247c2f74cd44a12a587f056979b798f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.778280 16060 raft_consensus.cc:2454] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2522fdb3bfeb457e9a395935914b896c in term 1.
I20240501 19:56:42.778280 15923 raft_consensus.cc:2454] T 0605625dbbe149538a96fde4aaf1d3ce P 3247c2f74cd44a12a587f056979b798f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2522fdb3bfeb457e9a395935914b896c in term 1.
I20240501 19:56:42.778563 16129 leader_election.cc:302] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2522fdb3bfeb457e9a395935914b896c, 3247c2f74cd44a12a587f056979b798f; no voters:
I20240501 19:56:42.778725 16344 raft_consensus.cc:2781] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:42.778942 16344 raft_consensus.cc:686] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [term 1 LEADER]: Becoming Leader. State: Replica: 2522fdb3bfeb457e9a395935914b896c, State: Running, Role: LEADER
I20240501 19:56:42.779104 16344 consensus_queue.cc:237] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } } peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } }
I20240501 19:56:42.779659 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.779408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.67:55147)
0501 19:56:42.779527 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.779580 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20240501 19:56:42.781087 15791 catalog_manager.cc:5484] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2522fdb3bfeb457e9a395935914b896c (127.8.168.67). New cstate: current_term: 1 leader_uuid: "2522fdb3bfeb457e9a395935914b896c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2522fdb3bfeb457e9a395935914b896c" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37273 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:42.810577 16246 tablet.cc:2369] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:42.811511 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.808761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.8.168.67:37273)
0501 19:56:42.808888 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:42.808892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:42.808909 (+ 17us) server_negotiation.cc:407] Connection header received
0501 19:56:42.808959 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:42.808963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:42.809015 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:42.809093 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:42.809415 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.810150 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:42.810849 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:42.811036 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:42.811085 (+ 49us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:42.811167 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:42.811239 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:42.811337 (+ 98us) server_negotiation.cc:299] Negotiation successful
0501 19:56:42.811386 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240501 19:56:42.813549 15923 raft_consensus.cc:1261] T 0605625dbbe149538a96fde4aaf1d3ce P 3247c2f74cd44a12a587f056979b798f [term 1 FOLLOWER]: Refusing update from remote peer 2522fdb3bfeb457e9a395935914b896c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.813552 16060 raft_consensus.cc:1261] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f [term 1 FOLLOWER]: Refusing update from remote peer 2522fdb3bfeb457e9a395935914b896c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.813997 16344 consensus_queue.cc:1035] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e74f065d34341d8a34c0b20761a558f" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.814163 16351 consensus_queue.cc:1035] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3247c2f74cd44a12a587f056979b798f" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.816073 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.815729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.66:35445)
0501 19:56:42.815929 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.815991 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20240501 19:56:42.816376 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.815758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.65:39493)
0501 19:56:42.816255 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.816305 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426}
I20240501 19:56:42.819759 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.819505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.67:57495)
0501 19:56:42.819624 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.819686 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20240501 19:56:42.819993 16354 mvcc.cc:205] Tried to move back new op lower bound from 7022974577919987712 to 7022974577785368576. Current Snapshot: MvccSnapshot[applied={T|T < 7022974577919987712 or (T in {7022974577919987712})}]
I20240501 19:56:42.820626 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.820381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.67:59711)
0501 19:56:42.820513 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.820560 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20240501 19:56:42.820801 16356 mvcc.cc:205] Tried to move back new op lower bound from 7022974577919987712 to 7022974577785368576. Current Snapshot: MvccSnapshot[applied={T|T < 7022974577919987712}]
I20240501 19:56:42.821821 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.821586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.65:46973)
0501 19:56:42.821697 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.821748 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20240501 19:56:42.823514 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.823245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45289 (local address 127.8.168.66:38553)
0501 19:56:42.823371 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.823419 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20240501 19:56:42.823868 15791 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37520:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:42.824018 15791 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37520:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:42.827811 15791 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 697109cc171e40c1bb4afb53014ca1bc: Sending DeleteTablet for 3 replicas of tablet 0605625dbbe149538a96fde4aaf1d3ce
I20240501 19:56:42.828364 15903 tablet_service.cc:1514] Processing DeleteTablet for tablet 0605625dbbe149538a96fde4aaf1d3ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:42 UTC) from {username='oryx', principal='oryx/127.8.168.124@KRBTEST.COM'} at 127.0.0.1:36228
I20240501 19:56:42.828599 16040 tablet_service.cc:1514] Processing DeleteTablet for tablet 0605625dbbe149538a96fde4aaf1d3ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:42 UTC) from {username='oryx', principal='oryx/127.8.168.124@KRBTEST.COM'} at 127.0.0.1:36830
I20240501 19:56:42.828874 16363 tablet_replica.cc:330] T 0605625dbbe149538a96fde4aaf1d3ce P 3247c2f74cd44a12a587f056979b798f: stopping tablet replica
I20240501 19:56:42.828917 16176 tablet_service.cc:1514] Processing DeleteTablet for tablet 0605625dbbe149538a96fde4aaf1d3ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:42 UTC) from {username='oryx', principal='oryx/127.8.168.124@KRBTEST.COM'} at 127.0.0.1:57926
I20240501 19:56:42.829030 16364 tablet_replica.cc:330] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f: stopping tablet replica
I20240501 19:56:42.829131 16364 raft_consensus.cc:2229] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:42.829133 16363 raft_consensus.cc:2229] T 0605625dbbe149538a96fde4aaf1d3ce P 3247c2f74cd44a12a587f056979b798f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:42.829262 16363 raft_consensus.cc:2258] T 0605625dbbe149538a96fde4aaf1d3ce P 3247c2f74cd44a12a587f056979b798f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:42.829262 16364 raft_consensus.cc:2258] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:42.829330 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 15749
I20240501 19:56:42.830914 16364 ts_tablet_manager.cc:1875] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:42.832780 16364 ts_tablet_manager.cc:1888] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:42.832859 16364 log.cc:1196] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/0605625dbbe149538a96fde4aaf1d3ce
I20240501 19:56:42.833037 16365 tablet_replica.cc:330] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c: stopping tablet replica
I20240501 19:56:42.833146 16364 ts_tablet_manager.cc:1909] T 0605625dbbe149538a96fde4aaf1d3ce P 2e74f065d34341d8a34c0b20761a558f: Deleting consensus metadata
I20240501 19:56:42.833158 16365 raft_consensus.cc:2229] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:42.833359 16365 raft_consensus.cc:2258] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:42.833964 16365 ts_tablet_manager.cc:1875] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:42.834033 15777 catalog_manager.cc:4830] TS 2e74f065d34341d8a34c0b20761a558f (127.8.168.66:41839): tablet 0605625dbbe149538a96fde4aaf1d3ce (table test-table [id=6e3cd111ea7f456c81e72302733d7157]) successfully deleted
I20240501 19:56:42.835741 16365 ts_tablet_manager.cc:1888] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:42.835816 16365 log.cc:1196] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/0605625dbbe149538a96fde4aaf1d3ce
I20240501 19:56:42.835803 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 15977
I20240501 19:56:42.836055 16365 ts_tablet_manager.cc:1909] T 0605625dbbe149538a96fde4aaf1d3ce P 2522fdb3bfeb457e9a395935914b896c: Deleting consensus metadata
I20240501 19:56:42.836522 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:42.835932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:46591 (local address 127.0.0.1:36232)
0501 19:56:42.836378 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:42.836426 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367}
W20240501 19:56:42.836678 15774 catalog_manager.cc:4573] TS 3247c2f74cd44a12a587f056979b798f (127.8.168.65:46591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0605625dbbe149538a96fde4aaf1d3ce: Network error: Client connection negotiation failed: client connection to 127.8.168.65:46591: connect: Connection refused (error 111)
I20240501 19:56:42.837559 15775 catalog_manager.cc:4830] TS 2522fdb3bfeb457e9a395935914b896c (127.8.168.67:37273): tablet 0605625dbbe149538a96fde4aaf1d3ce (table test-table [id=6e3cd111ea7f456c81e72302733d7157]) successfully deleted
I20240501 19:56:42.840878 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 16113
I20240501 19:56:42.846757 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 15465
I20240501 19:56:42.851416 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 15568
2024-05-01T19:56:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5060 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 19:56:42 dist-test-slave-tzct krb5kdc[16368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 19:56:42 dist-test-slave-tzct krb5kdc[16368](info): set up 2 sockets
May 01 19:56:42 dist-test-slave-tzct krb5kdc[16368](info): commencing operation
krb5kdc: starting...
W20240501 19:56:43.749423 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.877s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 19:56:43 dist-test-slave-tzct krb5kdc[16368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T19:56:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.126.keytab.
Entry for principal oryx/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.126.keytab.
I20240501 19:56:43.860411 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:35023
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.126.keytab
--principal=oryx/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:35023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:43.964166 16381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:43.964396 16381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:43.964455 16381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:43.967509 16381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:43.967579 16381 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240501 19:56:43.967595 16381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:43.967625 16381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:43.967658 16381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:43.967686 16381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:43.971885 16381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:35023
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.126.keytab
--principal=oryx/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:35023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:43.972792 16381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:43.973651 16381 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:43.980638 16386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:43.981236 16389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:43.981901 16387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:43.982347 16381 server_base.cc:1034] running on GCE node
I20240501 19:56:43.982829 16381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:43.983759 16381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:43.984903 16381 hybrid_clock.cc:648] HybridClock initialized: now 1714593403984893 us; error 27 us; skew 500 ppm
May 01 19:56:43 dist-test-slave-tzct krb5kdc[16368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593403, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:43.987527 16381 init.cc:377] Logged in from keytab as oryx/127.8.168.126@KRBTEST.COM (short username oryx)
I20240501 19:56:43.988448 16381 webserver.cc:460] Webserver started at http://127.8.168.126:35731/ using document root <none> and password file <none>
I20240501 19:56:43.989001 16381 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:43.989066 16381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:43.989248 16381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:43.990952 16381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "6fa2c4ee823246bca0fa71d967a21417"
format_stamp: "Formatted at 2024-05-01 19:56:43 on dist-test-slave-tzct"
server_key: "e8dd9017410aaec4ee568b998c131c01"
server_key_iv: "ea2a84ae4f28151cf4668dcf74e3a2b5"
server_key_version: "encryptionkey@0"
I20240501 19:56:43.991422 16381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "6fa2c4ee823246bca0fa71d967a21417"
format_stamp: "Formatted at 2024-05-01 19:56:43 on dist-test-slave-tzct"
server_key: "e8dd9017410aaec4ee568b998c131c01"
server_key_iv: "ea2a84ae4f28151cf4668dcf74e3a2b5"
server_key_version: "encryptionkey@0"
I20240501 19:56:43.994619 16381 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:43.996709 16396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:43.997713 16381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:43.997828 16381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "6fa2c4ee823246bca0fa71d967a21417"
format_stamp: "Formatted at 2024-05-01 19:56:43 on dist-test-slave-tzct"
server_key: "e8dd9017410aaec4ee568b998c131c01"
server_key_iv: "ea2a84ae4f28151cf4668dcf74e3a2b5"
server_key_version: "encryptionkey@0"
I20240501 19:56:43.997910 16381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:44.015990 16381 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:44.018942 16381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:44.019119 16381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:44.026768 16381 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:35023
I20240501 19:56:44.026779 16458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:35023 every 8 connection(s)
I20240501 19:56:44.027770 16381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:56:44.030277 16459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.035192 16459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417: Bootstrap starting.
I20240501 19:56:44.035952 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 16381
I20240501 19:56:44.036056 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:56:44.036262 8865 external_mini_cluster.cc:1428] Setting key c2f7ba3d6b2084eec47ca1b3a639362b
I20240501 19:56:44.037580 16459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.038252 16459 log.cc:824] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:44.040225 16459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417: No bootstrap required, opened a new log
May 01 19:56:44 dist-test-slave-tzct krb5kdc[16368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM
I20240501 19:56:44.043736 16459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa2c4ee823246bca0fa71d967a21417" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 35023 } }
I20240501 19:56:44.043947 16459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.044008 16459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa2c4ee823246bca0fa71d967a21417, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.044452 16459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa2c4ee823246bca0fa71d967a21417" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 35023 } }
I20240501 19:56:44.044569 16459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:44.044636 16459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:44.044754 16459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.045598 16459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa2c4ee823246bca0fa71d967a21417" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 35023 } }
I20240501 19:56:44.045897 16459 leader_election.cc:302] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fa2c4ee823246bca0fa71d967a21417; no voters:
I20240501 19:56:44.046149 16459 leader_election.cc:288] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:44.046291 16464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:44.046559 16464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [term 1 LEADER]: Becoming Leader. State: Replica: 6fa2c4ee823246bca0fa71d967a21417, State: Running, Role: LEADER
I20240501 19:56:44.046909 16464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa2c4ee823246bca0fa71d967a21417" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 35023 } }
I20240501 19:56:44.047199 16459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:44.048420 16465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fa2c4ee823246bca0fa71d967a21417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa2c4ee823246bca0fa71d967a21417" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 35023 } } }
I20240501 19:56:44.048527 16465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:44.048878 16466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fa2c4ee823246bca0fa71d967a21417. Latest consensus state: current_term: 1 leader_uuid: "6fa2c4ee823246bca0fa71d967a21417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa2c4ee823246bca0fa71d967a21417" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 35023 } } }
I20240501 19:56:44.048995 16466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:44.049237 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.037272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51954 (local address 127.8.168.126:35023)
0501 19:56:44.037732 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:44.037741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:44.037775 (+ 34us) server_negotiation.cc:407] Connection header received
0501 19:56:44.038382 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:44.038400 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:44.038721 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:44.039092 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:44.040076 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.040909 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.041750 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.041939 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.044340 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:44.044361 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:44.044372 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:44.044404 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:44.046312 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:44.046818 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:44.046823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:44.046828 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:44.046926 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:44.047191 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:44.047194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:44.047196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:44.047633 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:44.047852 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:44.048141 (+ 289us) server_negotiation.cc:299] Negotiation successful
0501 19:56:44.048430 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":178,"threads_started":1}
W20240501 19:56:44.051834 16479 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:56:44.051898 16479 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:56:44.051975 16474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:44.052937 16474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:44.057732 16474 catalog_manager.cc:1347] Generated new cluster ID: 1308466376604fa4a777f5e97df2bdbd
I20240501 19:56:44.057803 16474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:44.074143 16474 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:44.075438 16474 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:44.091176 16474 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417: Generated new TSK 0
I20240501 19:56:44.092041 16474 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.65.keytab.
Entry for principal oryx/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.65.keytab.
I20240501 19:56:44.119318 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:35023
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.65.keytab
--principal=oryx/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:44.223176 16485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:44.223394 16485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:44.223450 16485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:44.226419 16485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:44.226481 16485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:44.226529 16485 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 19:56:44.226581 16485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:56:44.230537 16485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.65.keytab
--principal=oryx/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.168.126:35023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:44.231341 16485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:44.232111 16485 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:44.241655 16490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:44.241765 16493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:44.242278 16491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:44.242802 16485 server_base.cc:1034] running on GCE node
I20240501 19:56:44.243208 16485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:44.243798 16485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:44.244978 16485 hybrid_clock.cc:648] HybridClock initialized: now 1714593404244965 us; error 29 us; skew 500 ppm
May 01 19:56:44 dist-test-slave-tzct krb5kdc[16368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593404, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:44.247479 16485 init.cc:377] Logged in from keytab as oryx/127.8.168.65@KRBTEST.COM (short username oryx)
I20240501 19:56:44.248340 16485 webserver.cc:460] Webserver started at http://127.8.168.65:39323/ using document root <none> and password file <none>
I20240501 19:56:44.248898 16485 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:44.248963 16485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:44.249135 16485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:44.250888 16485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "f30e604459744605bbdeb2fe62c9396b"
format_stamp: "Formatted at 2024-05-01 19:56:44 on dist-test-slave-tzct"
server_key: "137b9d066a1849de153eb81605a8bc53"
server_key_iv: "168ffe5b5ac6b4bd053b7813730f3c95"
server_key_version: "encryptionkey@0"
I20240501 19:56:44.251340 16485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "f30e604459744605bbdeb2fe62c9396b"
format_stamp: "Formatted at 2024-05-01 19:56:44 on dist-test-slave-tzct"
server_key: "137b9d066a1849de153eb81605a8bc53"
server_key_iv: "168ffe5b5ac6b4bd053b7813730f3c95"
server_key_version: "encryptionkey@0"
I20240501 19:56:44.254598 16485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:44.256867 16500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.257872 16485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:44.258013 16485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "f30e604459744605bbdeb2fe62c9396b"
format_stamp: "Formatted at 2024-05-01 19:56:44 on dist-test-slave-tzct"
server_key: "137b9d066a1849de153eb81605a8bc53"
server_key_iv: "168ffe5b5ac6b4bd053b7813730f3c95"
server_key_version: "encryptionkey@0"
I20240501 19:56:44.258109 16485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:44.278031 16485 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:44.280803 16485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:44.280963 16485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:44.282357 16485 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:44.282413 16485 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.282454 16485 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:44.282486 16485 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 19:56:44 dist-test-slave-tzct krb5kdc[16368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593404, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM
I20240501 19:56:44.291997 16485 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:45667
I20240501 19:56:44.292060 16615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:45667 every 8 connection(s)
I20240501 19:56:44.293140 16485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 19:56:44.294831 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 16485
I20240501 19:56:44.294894 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.283058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:45955 (local address 127.8.168.126:35023)
0501 19:56:44.283249 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:44.283253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:44.284047 (+ 794us) server_negotiation.cc:407] Connection header received
0501 19:56:44.285036 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:44.285040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:44.285098 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:44.285153 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:44.286945 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.287749 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.288620 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.288843 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.291344 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:44.291369 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:44.291372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:44.291403 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:44.293041 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:44.293507 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:44.293510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:44.293512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:44.293582 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:44.293895 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:44.293897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:44.293899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:44.294050 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:44.294148 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:44.294595 (+ 447us) server_negotiation.cc:299] Negotiation successful
0501 19:56:44.294734 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20240501 19:56:44.294940 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 19:56:44.295186 8865 external_mini_cluster.cc:1428] Setting key 3951b72c403263f43f14923c2f829679
I20240501 19:56:44.295440 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.283434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:35023 (local address 127.8.168.65:45955)
0501 19:56:44.283878 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:44.283919 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:44.284768 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:44.285565 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:44.285576 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:44.286021 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:44.286779 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.286797 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.287887 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.287891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:44.288483 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.288495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.288764 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.289551 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:44.289582 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:44.291183 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:44.293151 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:44.293155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:44.293164 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:44.293407 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:44.293690 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:44.293693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:44.293695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:44.293800 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:44.294161 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:44.294167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:44.294373 (+ 206us) client_negotiation.cc:769] Sending connection context
0501 19:56:44.294541 (+ 168us) client_negotiation.cc:240] Negotiation successful
0501 19:56:44.294793 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":117,"threads_started":1}
I20240501 19:56:44.296670 16616 heartbeater.cc:347] Connected to a master server at 127.8.168.126:35023
I20240501 19:56:44.296828 16616 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:44.297379 16616 heartbeater.cc:511] Master 127.8.168.126:35023 requested a full tablet report, sending...
I20240501 19:56:44.299057 16413 ts_manager.cc:194] Registered new tserver with Master: f30e604459744605bbdeb2fe62c9396b (127.8.168.65:45667)
I20240501 19:56:44.300208 16413 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:45955
WARNING: no policy specified for oryx/127.8.168.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.168.66@KRBTEST.COM" created.
I20240501 19:56:44.306839 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.298600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:35023 (local address 127.8.168.65:37041)
0501 19:56:44.298861 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:44.298875 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:44.298962 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:44.301123 (+ 2161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:44.301127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:44.301317 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:44.301627 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.301636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.302573 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.302577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:44.303137 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.303149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.303273 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.303808 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:44.303826 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:44.304137 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:44.305729 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:44.305732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:44.305734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:44.305883 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:44.306056 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:44.306059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:44.306061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:44.306114 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:44.306413 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:44.306416 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:44.306496 (+ 80us) client_negotiation.cc:769] Sending connection context
0501 19:56:44.306601 (+ 105us) client_negotiation.cc:240] Negotiation successful
0501 19:56:44.306698 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
I20240501 19:56:44.306931 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.300707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:37041 (local address 127.8.168.126:35023)
0501 19:56:44.300880 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:44.300883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:44.300896 (+ 13us) server_negotiation.cc:407] Connection header received
0501 19:56:44.300930 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:44.300933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:44.300975 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:44.301051 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:44.301757 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.302449 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.303253 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.303389 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.304249 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:44.304265 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:44.304268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:44.304292 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:44.305637 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:44.305955 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:44.305957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:44.305958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:44.305992 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:44.306191 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:44.306194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:44.306196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:44.306334 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:44.306407 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:44.306678 (+ 271us) server_negotiation.cc:299] Negotiation successful
0501 19:56:44.306783 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.66.keytab.
Entry for principal oryx/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.66.keytab.
I20240501 19:56:44.320266 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:35023
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.66.keytab
--principal=oryx/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:44.424711 16625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:44.424980 16625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:44.425038 16625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:44.428018 16625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:44.428076 16625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:44.428124 16625 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 19:56:44.428169 16625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:56:44.432200 16625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.66.keytab
--principal=oryx/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.168.126:35023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:44.433126 16625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:44.433915 16625 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:44.441413 16631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:44.442116 16633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:44.442442 16630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:44.443097 16625 server_base.cc:1034] running on GCE node
I20240501 19:56:44.443488 16625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:44.444034 16625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:44.445210 16625 hybrid_clock.cc:648] HybridClock initialized: now 1714593404445186 us; error 36 us; skew 500 ppm
May 01 19:56:44 dist-test-slave-tzct krb5kdc[16368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593404, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:44.447819 16625 init.cc:377] Logged in from keytab as oryx/127.8.168.66@KRBTEST.COM (short username oryx)
I20240501 19:56:44.448709 16625 webserver.cc:460] Webserver started at http://127.8.168.66:40013/ using document root <none> and password file <none>
I20240501 19:56:44.449240 16625 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:44.449294 16625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:44.449457 16625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:44.451170 16625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "e914b9526b3b43bda46fb11aeb9324b7"
format_stamp: "Formatted at 2024-05-01 19:56:44 on dist-test-slave-tzct"
server_key: "3ecd187a8b19619afd85c30ebbbfc702"
server_key_iv: "0ff53487c92365526cc296202e854f1a"
server_key_version: "encryptionkey@0"
I20240501 19:56:44.451632 16625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "e914b9526b3b43bda46fb11aeb9324b7"
format_stamp: "Formatted at 2024-05-01 19:56:44 on dist-test-slave-tzct"
server_key: "3ecd187a8b19619afd85c30ebbbfc702"
server_key_iv: "0ff53487c92365526cc296202e854f1a"
server_key_version: "encryptionkey@0"
I20240501 19:56:44.454871 16625 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240501 19:56:44.456990 16640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.457998 16625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:56:44.458110 16625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "e914b9526b3b43bda46fb11aeb9324b7"
format_stamp: "Formatted at 2024-05-01 19:56:44 on dist-test-slave-tzct"
server_key: "3ecd187a8b19619afd85c30ebbbfc702"
server_key_iv: "0ff53487c92365526cc296202e854f1a"
server_key_version: "encryptionkey@0"
I20240501 19:56:44.458191 16625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:44.477640 16625 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:44.481374 16625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:44.481532 16625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:44.482887 16625 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:44.482937 16625 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.482975 16625 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:44.483009 16625 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 19:56:44 dist-test-slave-tzct krb5kdc[16368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593404, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM
I20240501 19:56:44.492131 16625 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:34269
I20240501 19:56:44.492185 16755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:34269 every 8 connection(s)
I20240501 19:56:44.493046 16625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 19:56:44.495163 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.483680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:57243 (local address 127.8.168.126:35023)
0501 19:56:44.483795 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:44.483799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:44.484669 (+ 870us) server_negotiation.cc:407] Connection header received
0501 19:56:44.485680 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:44.485684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:44.485748 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:44.485808 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:44.487638 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.488404 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.489314 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.489504 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.491869 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:44.491890 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:44.491892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:44.491930 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:44.493438 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:44.493963 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:44.493966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:44.493967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:44.494012 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:44.494264 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:44.494267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:44.494269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:44.494431 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:44.494515 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:44.494913 (+ 398us) server_negotiation.cc:299] Negotiation successful
0501 19:56:44.495041 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240501 19:56:44.495636 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 16625
I20240501 19:56:44.495733 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.484057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:35023 (local address 127.8.168.66:57243)
0501 19:56:44.484533 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:44.484567 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:44.485418 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:44.486224 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:44.486236 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:44.486663 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:44.487447 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.487475 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.488534 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.488538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:44.489167 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.489177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.489394 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.489966 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:44.489991 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:44.491713 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:44.493557 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:44.493562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:44.493573 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:44.493867 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:44.494086 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:44.494089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:44.494090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:44.494165 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:44.494517 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:44.494521 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:44.494724 (+ 203us) client_negotiation.cc:769] Sending connection context
0501 19:56:44.494874 (+ 150us) client_negotiation.cc:240] Negotiation successful
0501 19:56:44.495082 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":141,"threads_started":1}
I20240501 19:56:44.495749 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 19:56:44.495990 8865 external_mini_cluster.cc:1428] Setting key 14e73250a1334bb0d7afe9249195ed28
I20240501 19:56:44.496994 16756 heartbeater.cc:347] Connected to a master server at 127.8.168.126:35023
I20240501 19:56:44.497123 16756 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:44.497682 16756 heartbeater.cc:511] Master 127.8.168.126:35023 requested a full tablet report, sending...
I20240501 19:56:44.498769 16411 ts_manager.cc:194] Registered new tserver with Master: e914b9526b3b43bda46fb11aeb9324b7 (127.8.168.66:34269)
I20240501 19:56:44.499495 16411 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:57243
WARNING: no policy specified for oryx/127.8.168.67@KRBTEST.COM; defaulting to no policy
I20240501 19:56:44.506639 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.499018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:35023 (local address 127.8.168.66:43221)
0501 19:56:44.500959 (+ 1941us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:44.500974 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:44.501087 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:44.501354 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:44.501356 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:44.501565 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:44.501861 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.501871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.502582 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.502586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:44.502957 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.502965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.503064 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.503652 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:44.503663 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:44.503930 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:44.505526 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:44.505529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:44.505531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:44.505685 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:44.505918 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:44.505920 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:44.505922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:44.505962 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:44.506236 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:44.506238 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:44.506300 (+ 62us) client_negotiation.cc:769] Sending connection context
0501 19:56:44.506372 (+ 72us) client_negotiation.cc:240] Negotiation successful
0501 19:56:44.506495 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1856,"thread_start_us":101,"threads_started":1}
I20240501 19:56:44.506651 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.499074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:43221 (local address 127.8.168.126:35023)
0501 19:56:44.499183 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:44.499186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:44.501050 (+ 1864us) server_negotiation.cc:407] Connection header received
0501 19:56:44.501188 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:44.501191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:44.501240 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:44.501321 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:44.501973 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.502477 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.503055 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.503237 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.504044 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:44.504059 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:44.504061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:44.504081 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:44.505415 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:44.505795 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:44.505799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:44.505801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:44.505846 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:44.506027 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:44.506029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:44.506030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:44.506174 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:44.506244 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:44.506429 (+ 185us) server_negotiation.cc:299] Negotiation successful
0501 19:56:44.506530 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.168.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.67.keytab.
Entry for principal oryx/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.67.keytab.
I20240501 19:56:44.520931 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:35023
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.67.keytab
--principal=oryx/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:44.626808 16765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:44.627044 16765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:44.627125 16765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:44.630170 16765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:44.630235 16765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:44.630306 16765 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240501 19:56:44.630367 16765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:56:44.634474 16765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.67.keytab
--principal=oryx/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.168.126:35023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:44.635380 16765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:44.636226 16765 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:44.643914 16771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:44.644335 16770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:44.644874 16773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:44.645334 16765 server_base.cc:1034] running on GCE node
I20240501 19:56:44.645659 16765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:44.646167 16765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:44.647312 16765 hybrid_clock.cc:648] HybridClock initialized: now 1714593404647284 us; error 41 us; skew 500 ppm
May 01 19:56:44 dist-test-slave-tzct krb5kdc[16368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593404, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:44.650079 16765 init.cc:377] Logged in from keytab as oryx/127.8.168.67@KRBTEST.COM (short username oryx)
I20240501 19:56:44.650969 16765 webserver.cc:460] Webserver started at http://127.8.168.67:41495/ using document root <none> and password file <none>
I20240501 19:56:44.651489 16765 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:44.651553 16765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:44.651726 16765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:44.653549 16765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "56d95af6390d40e7a395ff0a2d798726"
format_stamp: "Formatted at 2024-05-01 19:56:44 on dist-test-slave-tzct"
server_key: "84e6a37cb6a9ff37e2388632f8dce938"
server_key_iv: "3639d2d6cff3e2d5d2dd3291c58e4df5"
server_key_version: "encryptionkey@0"
I20240501 19:56:44.654014 16765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "56d95af6390d40e7a395ff0a2d798726"
format_stamp: "Formatted at 2024-05-01 19:56:44 on dist-test-slave-tzct"
server_key: "84e6a37cb6a9ff37e2388632f8dce938"
server_key_iv: "3639d2d6cff3e2d5d2dd3291c58e4df5"
server_key_version: "encryptionkey@0"
I20240501 19:56:44.657334 16765 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:44.659541 16780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.660504 16765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:44.660621 16765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "56d95af6390d40e7a395ff0a2d798726"
format_stamp: "Formatted at 2024-05-01 19:56:44 on dist-test-slave-tzct"
server_key: "84e6a37cb6a9ff37e2388632f8dce938"
server_key_iv: "3639d2d6cff3e2d5d2dd3291c58e4df5"
server_key_version: "encryptionkey@0"
I20240501 19:56:44.660709 16765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:44.678910 16765 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:44.681602 16765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:44.681763 16765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:44.683148 16765 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:44.683198 16765 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:44.683251 16765 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:44.683280 16765 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 19:56:44 dist-test-slave-tzct krb5kdc[16368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593404, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM
I20240501 19:56:44.692862 16765 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:34613
I20240501 19:56:44.692986 16895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:34613 every 8 connection(s)
I20240501 19:56:44.693765 16765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 19:56:44.695120 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.683883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:37743 (local address 127.8.168.126:35023)
0501 19:56:44.684037 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:44.684041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:44.684715 (+ 674us) server_negotiation.cc:407] Connection header received
0501 19:56:44.685566 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:44.685569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:44.685629 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:44.685704 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:44.687355 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.688125 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.688977 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.689190 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.691713 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:44.691733 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:44.691736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:44.691761 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:44.693375 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:44.693847 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:44.693850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:44.693851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:44.693908 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:44.694159 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:44.694163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:44.694165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:44.694334 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:44.694425 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:44.694892 (+ 467us) server_negotiation.cc:299] Negotiation successful
0501 19:56:44.694990 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20240501 19:56:44.695684 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.684145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:35023 (local address 127.8.168.67:37743)
0501 19:56:44.684573 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:44.684613 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:44.685361 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:44.685916 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:44.685928 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:44.686378 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:44.687180 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.687199 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.688257 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.688261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:44.688858 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.688868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.689080 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.689792 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:44.689932 (+ 140us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:44.691559 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:44.693484 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:44.693490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:44.693508 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:44.693748 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:44.693997 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:44.693999 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:44.694001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:44.694079 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:44.694432 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:44.694437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:44.694671 (+ 234us) client_negotiation.cc:769] Sending connection context
0501 19:56:44.694837 (+ 166us) client_negotiation.cc:240] Negotiation successful
0501 19:56:44.695037 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":108,"threads_started":1}
I20240501 19:56:44.695991 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 16765
I20240501 19:56:44.696120 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 19:56:44.696360 8865 external_mini_cluster.cc:1428] Setting key aecc89569c83d51dc812ac18d2f6c312
I20240501 19:56:44.697057 16896 heartbeater.cc:347] Connected to a master server at 127.8.168.126:35023
I20240501 19:56:44.697196 16896 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:44.697688 16896 heartbeater.cc:511] Master 127.8.168.126:35023 requested a full tablet report, sending...
I20240501 19:56:44.698617 16411 ts_manager.cc:194] Registered new tserver with Master: 56d95af6390d40e7a395ff0a2d798726 (127.8.168.67:34613)
I20240501 19:56:44.699318 16411 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:37743
I20240501 19:56:44.700013 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 19:56:44.706912 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.698806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:36689 (local address 127.8.168.126:35023)
0501 19:56:44.698925 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:44.698928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:44.699208 (+ 280us) server_negotiation.cc:407] Connection header received
0501 19:56:44.699312 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:44.699315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:44.699361 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:44.699434 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:44.701737 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.702424 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.703025 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.703208 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.704001 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:44.704037 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:44.704040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:44.704060 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:44.705603 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:44.705967 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:44.705970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:44.705971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:44.706015 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:44.706219 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:44.706222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:44.706224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:44.706357 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:44.706441 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:44.706681 (+ 240us) server_negotiation.cc:299] Negotiation successful
0501 19:56:44.706778 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240501 19:56:44.706928 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.698887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:35023 (local address 127.8.168.67:36689)
0501 19:56:44.699120 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:44.699133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:44.699231 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:44.699651 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:44.699654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:44.701353 (+ 1699us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:44.701628 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.701636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.702535 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.702538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:44.702895 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.702902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.703025 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.703578 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:44.703590 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:44.703894 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:44.705710 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:44.705713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:44.705715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:44.705870 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:44.706099 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:44.706101 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:44.706103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:44.706145 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:44.706442 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:44.706445 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:44.706518 (+ 73us) client_negotiation.cc:769] Sending connection context
0501 19:56:44.706614 (+ 96us) client_negotiation.cc:240] Negotiation successful
0501 19:56:44.706741 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
May 01 19:56:44 dist-test-slave-tzct krb5kdc[16368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593404, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 01 19:56:44 dist-test-slave-tzct krb5kdc[16368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:44 dist-test-slave-tzct krb5kdc[16368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
W20240501 19:56:44.722028 16462 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:44.717073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.8.168.126:35023)
0501 19:56:44.717191 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:44.717194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:44.717205 (+ 11us) server_negotiation.cc:407] Connection header received
0501 19:56:44.717243 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:44.717245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:44.717284 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:44.717355 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:44.718125 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.718606 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.719171 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.719285 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.721906 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51964: BlockingRecv error: recv got EOF from 127.0.0.1:51964 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
May 01 19:56:44 dist-test-slave-tzct krb5kdc[16368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593404, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM
I20240501 19:56:44.729838 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.723410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.8.168.126:35023)
0501 19:56:44.723528 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:44.723531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:44.723618 (+ 87us) server_negotiation.cc:407] Connection header received
0501 19:56:44.723723 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:44.723725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:44.723762 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:44.723828 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:44.724402 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.724905 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.725428 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.725556 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.727512 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:44.727527 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:44.727530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:44.727553 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:44.728833 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:44.729120 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:44.729123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:44.729124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:44.729157 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:44.729288 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:44.729290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:44.729291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:44.729413 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:44.729515 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:44.729645 (+ 130us) server_negotiation.cc:299] Negotiation successful
0501 19:56:44.729721 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20240501 19:56:44.732367 16413 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:56:44.734853 16413 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:44.746816 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.743179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:34269 (local address 127.0.0.1:42920)
0501 19:56:44.743562 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:44.743575 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:44.743659 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:44.744407 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:44.744410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:44.744424 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:44.744622 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.744629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.745590 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.745593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:44.746338 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.746347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.746447 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.746500 (+ 53us) client_negotiation.cc:769] Sending connection context
0501 19:56:44.746597 (+ 97us) client_negotiation.cc:240] Negotiation successful
0501 19:56:44.746674 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":99,"threads_started":1}
I20240501 19:56:44.747115 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.742613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:34613 (local address 127.0.0.1:42818)
0501 19:56:44.742833 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:44.742876 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:44.743012 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:44.744151 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:44.744157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:44.744178 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:44.744484 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.744495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.745931 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.745935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:44.746767 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.746776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.746866 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.746882 (+ 16us) client_negotiation.cc:769] Sending connection context
0501 19:56:44.746928 (+ 46us) client_negotiation.cc:240] Negotiation successful
0501 19:56:44.746980 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"spinlock_wait_cycles":1408,"thread_start_us":88,"threads_started":1}
I20240501 19:56:44.747105 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.742507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:45667 (local address 127.0.0.1:43730)
0501 19:56:44.742833 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:44.742876 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:44.743012 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:44.744313 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:44.744315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:44.744331 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:44.744563 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.744571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.745976 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.745980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:44.746730 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.746740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.746838 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.746854 (+ 16us) client_negotiation.cc:769] Sending connection context
0501 19:56:44.746898 (+ 44us) client_negotiation.cc:240] Negotiation successful
0501 19:56:44.746951 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"spinlock_wait_cycles":1920,"thread_start_us":90,"threads_started":1}
I20240501 19:56:44.747785 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.743241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.8.168.65:45667)
0501 19:56:44.743663 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:44.743669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:44.743692 (+ 23us) server_negotiation.cc:407] Connection header received
0501 19:56:44.743782 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:44.743788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:44.743969 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:44.744166 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:44.744663 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.745837 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.746845 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.747350 (+ 505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.747523 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:44.747589 (+ 66us) server_negotiation.cc:299] Negotiation successful
0501 19:56:44.747664 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":88,"threads_started":1}
I20240501 19:56:44.748086 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.743271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42818 (local address 127.8.168.67:34613)
0501 19:56:44.743783 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:44.743788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:44.743805 (+ 17us) server_negotiation.cc:407] Connection header received
0501 19:56:44.743875 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:44.743879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:44.743994 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:44.744152 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:44.744610 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.745818 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.747185 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.747755 (+ 570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.747830 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:44.747898 (+ 68us) server_negotiation.cc:299] Negotiation successful
0501 19:56:44.747975 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":91,"threads_started":1}
I20240501 19:56:44.748970 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.743389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42920 (local address 127.8.168.66:34269)
0501 19:56:44.744011 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:44.744018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:44.744039 (+ 21us) server_negotiation.cc:407] Connection header received
0501 19:56:44.744115 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:44.744120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:44.744269 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:44.744411 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:44.744755 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.745480 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.747488 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.748587 (+ 1099us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.748682 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:44.748767 (+ 85us) server_negotiation.cc:299] Negotiation successful
0501 19:56:44.748843 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":99,"threads_started":1}
I20240501 19:56:44.750319 16550 tablet_service.cc:1467] Processing CreateTablet for tablet c857d9fc4e3a4302934382c641a0bdaa (DEFAULT_TABLE table=test-table [id=5878694ce98a4d94be49cfaf3224da40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.750315 16830 tablet_service.cc:1467] Processing CreateTablet for tablet c857d9fc4e3a4302934382c641a0bdaa (DEFAULT_TABLE table=test-table [id=5878694ce98a4d94be49cfaf3224da40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.751134 16550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c857d9fc4e3a4302934382c641a0bdaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.751168 16830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c857d9fc4e3a4302934382c641a0bdaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.750998 16688 tablet_service.cc:1467] Processing CreateTablet for tablet c857d9fc4e3a4302934382c641a0bdaa (DEFAULT_TABLE table=test-table [id=5878694ce98a4d94be49cfaf3224da40]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.751745 16688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c857d9fc4e3a4302934382c641a0bdaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.756233 16924 tablet_bootstrap.cc:492] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b: Bootstrap starting.
I20240501 19:56:44.757465 16926 tablet_bootstrap.cc:492] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726: Bootstrap starting.
I20240501 19:56:44.757679 16925 tablet_bootstrap.cc:492] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7: Bootstrap starting.
I20240501 19:56:44.758826 16924 tablet_bootstrap.cc:654] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.759477 16924 log.cc:824] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:44.759868 16926 tablet_bootstrap.cc:654] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.759868 16925 tablet_bootstrap.cc:654] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.760623 16925 log.cc:824] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:44.760623 16926 log.cc:824] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:44.761082 16924 tablet_bootstrap.cc:492] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b: No bootstrap required, opened a new log
I20240501 19:56:44.761265 16924 ts_tablet_manager.cc:1367] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:44.762383 16926 tablet_bootstrap.cc:492] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726: No bootstrap required, opened a new log
I20240501 19:56:44.762619 16926 ts_tablet_manager.cc:1367] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240501 19:56:44.762805 16925 tablet_bootstrap.cc:492] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7: No bootstrap required, opened a new log
I20240501 19:56:44.762960 16925 ts_tablet_manager.cc:1367] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 19:56:44.763984 16924 raft_consensus.cc:348] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } }
I20240501 19:56:44.764292 16924 raft_consensus.cc:374] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.764394 16924 raft_consensus.cc:729] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f30e604459744605bbdeb2fe62c9396b, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.764889 16924 consensus_queue.cc:260] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } }
I20240501 19:56:44.765625 16616 heartbeater.cc:503] Master 127.8.168.126:35023 was elected leader, sending a full tablet report...
I20240501 19:56:44.765908 16926 raft_consensus.cc:348] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } }
I20240501 19:56:44.765908 16925 raft_consensus.cc:348] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } }
I20240501 19:56:44.766111 16926 raft_consensus.cc:374] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.766111 16925 raft_consensus.cc:374] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.766163 16925 raft_consensus.cc:729] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e914b9526b3b43bda46fb11aeb9324b7, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.766170 16926 raft_consensus.cc:729] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d95af6390d40e7a395ff0a2d798726, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.766578 16925 consensus_queue.cc:260] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } }
I20240501 19:56:44.766582 16926 consensus_queue.cc:260] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } }
I20240501 19:56:44.766904 16924 ts_tablet_manager.cc:1398] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 19:56:44.767397 16925 ts_tablet_manager.cc:1398] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:56:44.767397 16926 ts_tablet_manager.cc:1398] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:44.767773 16756 heartbeater.cc:503] Master 127.8.168.126:35023 was elected leader, sending a full tablet report...
I20240501 19:56:44.767899 16896 heartbeater.cc:503] Master 127.8.168.126:35023 was elected leader, sending a full tablet report...
W20240501 19:56:44.794211 16617 tablet.cc:2369] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:44.916972 16930 raft_consensus.cc:482] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:44.917191 16930 raft_consensus.cc:504] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } }
I20240501 19:56:44.918262 16930 leader_election.cc:288] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56d95af6390d40e7a395ff0a2d798726 (127.8.168.67:34613), e914b9526b3b43bda46fb11aeb9324b7 (127.8.168.66:34269)
I20240501 19:56:44.921769 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.918518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:34613 (local address 127.8.168.65:50959)
0501 19:56:44.918800 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:44.918815 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:44.918942 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:44.919255 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:44.919258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:44.919272 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:44.919503 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.919511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.920429 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.920431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:44.921317 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.921328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.921454 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.921477 (+ 23us) client_negotiation.cc:769] Sending connection context
0501 19:56:44.921529 (+ 52us) client_negotiation.cc:240] Negotiation successful
0501 19:56:44.921588 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":118,"threads_started":1}
I20240501 19:56:44.921769 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.918903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:34269 (local address 127.8.168.65:47131)
0501 19:56:44.919184 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:44.919195 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:44.919266 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:44.919502 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:44.919505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:44.919520 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:44.919719 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.919725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.920579 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.920582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:44.921378 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:44.921386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.921477 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.921489 (+ 12us) client_negotiation.cc:769] Sending connection context
0501 19:56:44.921533 (+ 44us) client_negotiation.cc:240] Negotiation successful
0501 19:56:44.921587 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":61,"threads_started":1}
I20240501 19:56:44.922188 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.918646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:50959 (local address 127.8.168.67:34613)
0501 19:56:44.918801 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:44.918805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:44.918894 (+ 89us) server_negotiation.cc:407] Connection header received
0501 19:56:44.919080 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:44.919084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:44.919140 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:44.919225 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:44.919618 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.920324 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.921432 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.921901 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.921941 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:44.921995 (+ 54us) server_negotiation.cc:299] Negotiation successful
0501 19:56:44.922049 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20240501 19:56:44.922168 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.919081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:47131 (local address 127.8.168.66:34269)
0501 19:56:44.919227 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:44.919231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:44.919245 (+ 14us) server_negotiation.cc:407] Connection header received
0501 19:56:44.919366 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:44.919369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:44.919406 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:44.919462 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:44.919821 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.920491 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.921471 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.921927 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.921958 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:44.922000 (+ 42us) server_negotiation.cc:299] Negotiation successful
0501 19:56:44.922048 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20240501 19:56:44.922732 16850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c857d9fc4e3a4302934382c641a0bdaa" candidate_uuid: "f30e604459744605bbdeb2fe62c9396b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56d95af6390d40e7a395ff0a2d798726" is_pre_election: true
I20240501 19:56:44.922745 16710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c857d9fc4e3a4302934382c641a0bdaa" candidate_uuid: "f30e604459744605bbdeb2fe62c9396b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e914b9526b3b43bda46fb11aeb9324b7" is_pre_election: true
I20240501 19:56:44.923015 16710 raft_consensus.cc:2454] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f30e604459744605bbdeb2fe62c9396b in term 0.
I20240501 19:56:44.923017 16850 raft_consensus.cc:2454] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f30e604459744605bbdeb2fe62c9396b in term 0.
I20240501 19:56:44.923439 16501 leader_election.cc:302] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56d95af6390d40e7a395ff0a2d798726, f30e604459744605bbdeb2fe62c9396b; no voters:
I20240501 19:56:44.923679 16930 raft_consensus.cc:2781] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:44.923746 16930 raft_consensus.cc:482] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:44.923786 16930 raft_consensus.cc:3037] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.924698 16930 raft_consensus.cc:504] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } }
I20240501 19:56:44.925052 16930 leader_election.cc:288] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [CANDIDATE]: Term 1 election: Requested vote from peers 56d95af6390d40e7a395ff0a2d798726 (127.8.168.67:34613), e914b9526b3b43bda46fb11aeb9324b7 (127.8.168.66:34269)
I20240501 19:56:44.925391 16850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c857d9fc4e3a4302934382c641a0bdaa" candidate_uuid: "f30e604459744605bbdeb2fe62c9396b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56d95af6390d40e7a395ff0a2d798726"
I20240501 19:56:44.925436 16710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c857d9fc4e3a4302934382c641a0bdaa" candidate_uuid: "f30e604459744605bbdeb2fe62c9396b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e914b9526b3b43bda46fb11aeb9324b7"
I20240501 19:56:44.925504 16850 raft_consensus.cc:3037] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.925529 16710 raft_consensus.cc:3037] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.926414 16710 raft_consensus.cc:2454] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f30e604459744605bbdeb2fe62c9396b in term 1.
I20240501 19:56:44.926417 16850 raft_consensus.cc:2454] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f30e604459744605bbdeb2fe62c9396b in term 1.
I20240501 19:56:44.926709 16501 leader_election.cc:302] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e914b9526b3b43bda46fb11aeb9324b7, f30e604459744605bbdeb2fe62c9396b; no voters:
I20240501 19:56:44.926883 16930 raft_consensus.cc:2781] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:44.927064 16930 raft_consensus.cc:686] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [term 1 LEADER]: Becoming Leader. State: Replica: f30e604459744605bbdeb2fe62c9396b, State: Running, Role: LEADER
I20240501 19:56:44.927379 16930 consensus_queue.cc:237] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } }
I20240501 19:56:44.930372 16412 catalog_manager.cc:5484] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b reported cstate change: term changed from 0 to 1, leader changed from <none> to f30e604459744605bbdeb2fe62c9396b (127.8.168.65). New cstate: current_term: 1 leader_uuid: "f30e604459744605bbdeb2fe62c9396b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:44.944968 16897 tablet.cc:2369] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:44.981038 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:44.978047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.8.168.65:45667)
0501 19:56:44.978199 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:44.978204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:44.978218 (+ 14us) server_negotiation.cc:407] Connection header received
0501 19:56:44.978300 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:44.978303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:44.978355 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:44.978451 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:44.978778 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.979264 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:44.979974 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:44.980150 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:44.980207 (+ 57us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:44.980638 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:44.980770 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:44.980852 (+ 82us) server_negotiation.cc:299] Negotiation successful
0501 19:56:44.980902 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20240501 19:56:44.986585 16850 raft_consensus.cc:1261] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726 [term 1 FOLLOWER]: Refusing update from remote peer f30e604459744605bbdeb2fe62c9396b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:44.986596 16710 raft_consensus.cc:1261] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7 [term 1 FOLLOWER]: Refusing update from remote peer f30e604459744605bbdeb2fe62c9396b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:44.987243 16935 consensus_queue.cc:1035] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [LEADER]: Connected to new peer: Peer: permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:44.987484 16930 consensus_queue.cc:1035] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:56:44.994087 16757 tablet.cc:2369] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:44.994514 16938 mvcc.cc:205] Tried to move back new op lower bound from 7022974586816745472 to 7022974586585374720. Current Snapshot: MvccSnapshot[applied={T|T < 7022974586816745472}]
I20240501 19:56:44.996027 16939 mvcc.cc:205] Tried to move back new op lower bound from 7022974586816745472 to 7022974586585374720. Current Snapshot: MvccSnapshot[applied={T|T < 7022974586816745472}]
I20240501 19:56:45.002610 16411 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51980:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:45.002781 16411 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51980:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:45.005347 16411 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417: Sending DeleteTablet for 3 replicas of tablet c857d9fc4e3a4302934382c641a0bdaa
I20240501 19:56:45.006383 16830 tablet_service.cc:1514] Processing DeleteTablet for tablet c857d9fc4e3a4302934382c641a0bdaa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:44 UTC) from {username='oryx', principal='oryx/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:42818
I20240501 19:56:45.006388 16688 tablet_service.cc:1514] Processing DeleteTablet for tablet c857d9fc4e3a4302934382c641a0bdaa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:44 UTC) from {username='oryx', principal='oryx/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:42920
I20240501 19:56:45.006852 16947 tablet_replica.cc:330] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7: stopping tablet replica
I20240501 19:56:45.006975 16550 tablet_service.cc:1514] Processing DeleteTablet for tablet c857d9fc4e3a4302934382c641a0bdaa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:44 UTC) from {username='oryx', principal='oryx/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:43730
I20240501 19:56:45.007164 16947 raft_consensus.cc:2229] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.007476 16947 raft_consensus.cc:2258] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.007555 16948 tablet_replica.cc:330] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726: stopping tablet replica
I20240501 19:56:45.007548 16411 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:56:45.007951 16411 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:45.007999 16949 tablet_replica.cc:330] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b: stopping tablet replica
I20240501 19:56:45.008173 16949 raft_consensus.cc:2229] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:45.008430 16949 raft_consensus.cc:2258] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.008680 16948 raft_consensus.cc:2229] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.008986 16948 raft_consensus.cc:2258] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.009923 16947 ts_tablet_manager.cc:1875] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.010326 16949 ts_tablet_manager.cc:1875] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.012398 16947 ts_tablet_manager.cc:1888] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:45.012501 16947 log.cc:1196] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/c857d9fc4e3a4302934382c641a0bdaa
I20240501 19:56:45.012614 16949 ts_tablet_manager.cc:1888] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:45.012714 16949 log.cc:1196] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/wal/wals/c857d9fc4e3a4302934382c641a0bdaa
I20240501 19:56:45.012845 16947 ts_tablet_manager.cc:1909] T c857d9fc4e3a4302934382c641a0bdaa P e914b9526b3b43bda46fb11aeb9324b7: Deleting consensus metadata
I20240501 19:56:45.013013 16949 ts_tablet_manager.cc:1909] T c857d9fc4e3a4302934382c641a0bdaa P f30e604459744605bbdeb2fe62c9396b: Deleting consensus metadata
I20240501 19:56:45.013780 16550 tablet_service.cc:1467] Processing CreateTablet for tablet e08ba3cfd7ba408f8bd6caf4891db9a2 (DEFAULT_TABLE table=test-table [id=78557681920c4e4b8bfedb056b01fdec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.014030 16550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e08ba3cfd7ba408f8bd6caf4891db9a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.015986 16924 tablet_bootstrap.cc:492] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b: Bootstrap starting.
I20240501 19:56:45.016137 16398 catalog_manager.cc:4830] TS f30e604459744605bbdeb2fe62c9396b (127.8.168.65:45667): tablet c857d9fc4e3a4302934382c641a0bdaa (table test-table [id=5878694ce98a4d94be49cfaf3224da40]) successfully deleted
I20240501 19:56:45.016316 16397 catalog_manager.cc:4830] TS e914b9526b3b43bda46fb11aeb9324b7 (127.8.168.66:34269): tablet c857d9fc4e3a4302934382c641a0bdaa (table test-table [id=5878694ce98a4d94be49cfaf3224da40]) successfully deleted
I20240501 19:56:45.016882 16924 tablet_bootstrap.cc:654] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.016899 16830 tablet_service.cc:1467] Processing CreateTablet for tablet e08ba3cfd7ba408f8bd6caf4891db9a2 (DEFAULT_TABLE table=test-table [id=78557681920c4e4b8bfedb056b01fdec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.017232 16830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e08ba3cfd7ba408f8bd6caf4891db9a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.017222 16688 tablet_service.cc:1467] Processing CreateTablet for tablet e08ba3cfd7ba408f8bd6caf4891db9a2 (DEFAULT_TABLE table=test-table [id=78557681920c4e4b8bfedb056b01fdec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.017449 16688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e08ba3cfd7ba408f8bd6caf4891db9a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.018517 16924 tablet_bootstrap.cc:492] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b: No bootstrap required, opened a new log
I20240501 19:56:45.018589 16924 ts_tablet_manager.cc:1367] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:45.019006 16924 raft_consensus.cc:348] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.019101 16924 raft_consensus.cc:374] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.019129 16924 raft_consensus.cc:729] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f30e604459744605bbdeb2fe62c9396b, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.019243 16924 consensus_queue.cc:260] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.019424 16925 tablet_bootstrap.cc:492] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7: Bootstrap starting.
I20240501 19:56:45.019466 16926 tablet_bootstrap.cc:492] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726: Bootstrap starting.
I20240501 19:56:45.019511 16924 ts_tablet_manager.cc:1398] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:45.020289 16925 tablet_bootstrap.cc:654] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.020478 16926 tablet_bootstrap.cc:654] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.021364 16948 ts_tablet_manager.cc:1875] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.022416 16925 tablet_bootstrap.cc:492] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7: No bootstrap required, opened a new log
I20240501 19:56:45.022495 16925 ts_tablet_manager.cc:1367] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:56:45.023041 16925 raft_consensus.cc:348] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.023135 16925 raft_consensus.cc:374] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.023164 16925 raft_consensus.cc:729] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e914b9526b3b43bda46fb11aeb9324b7, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.023293 16925 consensus_queue.cc:260] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.023540 16925 ts_tablet_manager.cc:1398] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:45.023681 16948 ts_tablet_manager.cc:1888] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:45.023761 16948 log.cc:1196] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/c857d9fc4e3a4302934382c641a0bdaa
I20240501 19:56:45.023977 16948 ts_tablet_manager.cc:1909] T c857d9fc4e3a4302934382c641a0bdaa P 56d95af6390d40e7a395ff0a2d798726: Deleting consensus metadata
I20240501 19:56:45.025053 16397 catalog_manager.cc:4830] TS 56d95af6390d40e7a395ff0a2d798726 (127.8.168.67:34613): tablet c857d9fc4e3a4302934382c641a0bdaa (table test-table [id=5878694ce98a4d94be49cfaf3224da40]) successfully deleted
I20240501 19:56:45.025601 16926 tablet_bootstrap.cc:492] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726: No bootstrap required, opened a new log
I20240501 19:56:45.025669 16926 ts_tablet_manager.cc:1367] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:45.026016 16926 raft_consensus.cc:348] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.026103 16926 raft_consensus.cc:374] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.026131 16926 raft_consensus.cc:729] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d95af6390d40e7a395ff0a2d798726, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.026257 16926 consensus_queue.cc:260] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.026492 16926 ts_tablet_manager.cc:1398] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:45.092980 16935 raft_consensus.cc:482] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.093140 16935 raft_consensus.cc:504] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.093595 16935 leader_election.cc:288] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e914b9526b3b43bda46fb11aeb9324b7 (127.8.168.66:34269), 56d95af6390d40e7a395ff0a2d798726 (127.8.168.67:34613)
I20240501 19:56:45.094048 16710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e08ba3cfd7ba408f8bd6caf4891db9a2" candidate_uuid: "f30e604459744605bbdeb2fe62c9396b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e914b9526b3b43bda46fb11aeb9324b7" is_pre_election: true
I20240501 19:56:45.094094 16850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e08ba3cfd7ba408f8bd6caf4891db9a2" candidate_uuid: "f30e604459744605bbdeb2fe62c9396b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56d95af6390d40e7a395ff0a2d798726" is_pre_election: true
I20240501 19:56:45.094197 16710 raft_consensus.cc:2454] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f30e604459744605bbdeb2fe62c9396b in term 0.
I20240501 19:56:45.094228 16850 raft_consensus.cc:2454] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f30e604459744605bbdeb2fe62c9396b in term 0.
I20240501 19:56:45.094550 16501 leader_election.cc:302] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e914b9526b3b43bda46fb11aeb9324b7, f30e604459744605bbdeb2fe62c9396b; no voters:
I20240501 19:56:45.094700 16935 raft_consensus.cc:2781] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:45.094761 16935 raft_consensus.cc:482] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:45.094798 16935 raft_consensus.cc:3037] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.095680 16935 raft_consensus.cc:504] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.096007 16935 leader_election.cc:288] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [CANDIDATE]: Term 1 election: Requested vote from peers e914b9526b3b43bda46fb11aeb9324b7 (127.8.168.66:34269), 56d95af6390d40e7a395ff0a2d798726 (127.8.168.67:34613)
I20240501 19:56:45.096343 16710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e08ba3cfd7ba408f8bd6caf4891db9a2" candidate_uuid: "f30e604459744605bbdeb2fe62c9396b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e914b9526b3b43bda46fb11aeb9324b7"
I20240501 19:56:45.096431 16850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e08ba3cfd7ba408f8bd6caf4891db9a2" candidate_uuid: "f30e604459744605bbdeb2fe62c9396b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56d95af6390d40e7a395ff0a2d798726"
I20240501 19:56:45.096470 16710 raft_consensus.cc:3037] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.096550 16850 raft_consensus.cc:3037] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.097430 16710 raft_consensus.cc:2454] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f30e604459744605bbdeb2fe62c9396b in term 1.
I20240501 19:56:45.097625 16850 raft_consensus.cc:2454] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f30e604459744605bbdeb2fe62c9396b in term 1.
I20240501 19:56:45.097723 16501 leader_election.cc:302] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e914b9526b3b43bda46fb11aeb9324b7, f30e604459744605bbdeb2fe62c9396b; no voters:
I20240501 19:56:45.097870 16935 raft_consensus.cc:2781] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:45.097963 16935 raft_consensus.cc:686] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [term 1 LEADER]: Becoming Leader. State: Replica: f30e604459744605bbdeb2fe62c9396b, State: Running, Role: LEADER
I20240501 19:56:45.098098 16935 consensus_queue.cc:237] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.099804 16411 catalog_manager.cc:5484] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b reported cstate change: term changed from 0 to 1, leader changed from <none> to f30e604459744605bbdeb2fe62c9396b (127.8.168.65). New cstate: current_term: 1 leader_uuid: "f30e604459744605bbdeb2fe62c9396b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } health_report { overall_health: UNKNOWN } } }
May 01 19:56:45 dist-test-slave-tzct krb5kdc[16368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593403, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.126@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM
I20240501 19:56:45.142138 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.134576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:35023 (local address 127.0.0.1:51988)
0501 19:56:45.134849 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:45.134861 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:45.134975 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:45.135211 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:45.135214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:45.135643 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:45.135841 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.135848 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.136593 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.136596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:45.137011 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.137020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.137123 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.137709 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:45.137723 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:45.139259 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:45.140782 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:45.140790 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:45.140795 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:45.140993 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:45.141192 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:45.141195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:45.141197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:45.141238 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:45.141537 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:45.141544 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:45.141632 (+ 88us) client_negotiation.cc:769] Sending connection context
0501 19:56:45.141726 (+ 94us) client_negotiation.cc:240] Negotiation successful
0501 19:56:45.141873 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":131,"threads_started":1}
I20240501 19:56:45.142138 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.134614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.8.168.126:35023)
0501 19:56:45.134729 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:45.134733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:45.134931 (+ 198us) server_negotiation.cc:407] Connection header received
0501 19:56:45.135075 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:45.135078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:45.135117 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:45.135180 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:45.135984 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.136497 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.137109 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.137267 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.139366 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:45.139381 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:45.139390 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:45.139410 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:45.140664 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:45.141077 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:45.141080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:45.141082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:45.141119 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:45.141306 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:45.141309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:45.141311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:45.141444 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:45.141524 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:45.141781 (+ 257us) server_negotiation.cc:299] Negotiation successful
0501 19:56:45.141941 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":42}
I20240501 19:56:45.151085 16411 catalog_manager.cc:1904] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:51988:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240501 19:56:45.151783 16411 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:45.156424 16550 tablet_service.cc:1467] Processing CreateTablet for tablet 2eee4f3079284eabbeba69d24e500dcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bcca29c60333442fa72b90699d45f95f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 19:56:45.156630 16830 tablet_service.cc:1467] Processing CreateTablet for tablet 2eee4f3079284eabbeba69d24e500dcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bcca29c60333442fa72b90699d45f95f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 19:56:45.156836 16550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eee4f3079284eabbeba69d24e500dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.156569 16688 tablet_service.cc:1467] Processing CreateTablet for tablet 2eee4f3079284eabbeba69d24e500dcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bcca29c60333442fa72b90699d45f95f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240501 19:56:45.157006 16830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eee4f3079284eabbeba69d24e500dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.157135 16688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eee4f3079284eabbeba69d24e500dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.159396 16924 tablet_bootstrap.cc:492] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b: Bootstrap starting.
I20240501 19:56:45.159528 16926 tablet_bootstrap.cc:492] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726: Bootstrap starting.
I20240501 19:56:45.159957 16925 tablet_bootstrap.cc:492] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7: Bootstrap starting.
I20240501 19:56:45.160302 16926 tablet_bootstrap.cc:654] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.160302 16924 tablet_bootstrap.cc:654] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.160827 16925 tablet_bootstrap.cc:654] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.162032 16926 tablet_bootstrap.cc:492] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726: No bootstrap required, opened a new log
I20240501 19:56:45.162130 16926 ts_tablet_manager.cc:1367] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:45.162390 16924 tablet_bootstrap.cc:492] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b: No bootstrap required, opened a new log
I20240501 19:56:45.162482 16924 ts_tablet_manager.cc:1367] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:45.162613 16926 raft_consensus.cc:348] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.162724 16926 raft_consensus.cc:374] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.162760 16926 raft_consensus.cc:729] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d95af6390d40e7a395ff0a2d798726, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.162860 16924 raft_consensus.cc:348] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.162889 16926 consensus_queue.cc:260] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.162950 16924 raft_consensus.cc:374] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.162981 16924 raft_consensus.cc:729] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f30e604459744605bbdeb2fe62c9396b, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.163080 16924 consensus_queue.cc:260] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.163148 16926 ts_tablet_manager.cc:1398] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:45.163209 16925 tablet_bootstrap.cc:492] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7: No bootstrap required, opened a new log
I20240501 19:56:45.163278 16925 ts_tablet_manager.cc:1367] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:45.163306 16924 ts_tablet_manager.cc:1398] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:45.163445 16931 tablet_replica.cc:439] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } }
I20240501 19:56:45.163592 16931 tablet_replica.cc:442] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:45.163651 16930 tablet_replica.cc:439] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } }
I20240501 19:56:45.163746 16930 tablet_replica.cc:442] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:45.163686 16925 raft_consensus.cc:348] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.163781 16925 raft_consensus.cc:374] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.163810 16925 raft_consensus.cc:729] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e914b9526b3b43bda46fb11aeb9324b7, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.163915 16925 consensus_queue.cc:260] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.164162 16925 ts_tablet_manager.cc:1398] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:45.164500 16932 tablet_replica.cc:439] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } }
I20240501 19:56:45.164602 16932 tablet_replica.cc:442] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:45.340904 16932 raft_consensus.cc:482] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.341053 16932 raft_consensus.cc:504] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.342126 16932 leader_election.cc:288] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f30e604459744605bbdeb2fe62c9396b (127.8.168.65:45667), 56d95af6390d40e7a395ff0a2d798726 (127.8.168.67:34613)
I20240501 19:56:45.345036 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.342396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:34613 (local address 127.8.168.66:54611)
0501 19:56:45.342700 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:45.342712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:45.342815 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:45.343071 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:45.343074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:45.343087 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:45.343286 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.343291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.343960 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.343963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:45.344563 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.344571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.344669 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.344690 (+ 21us) client_negotiation.cc:769] Sending connection context
0501 19:56:45.344771 (+ 81us) client_negotiation.cc:240] Negotiation successful
0501 19:56:45.344825 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":72,"threads_started":1}
I20240501 19:56:45.345038 16964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.342371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:45667 (local address 127.8.168.66:37655)
0501 19:56:45.342623 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:45.342638 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:45.342741 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:45.342997 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:45.343000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:45.343016 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:45.343260 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.343267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.343958 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.343960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:45.344559 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.344567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.344668 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.344690 (+ 22us) client_negotiation.cc:769] Sending connection context
0501 19:56:45.344771 (+ 81us) client_negotiation.cc:240] Negotiation successful
0501 19:56:45.344828 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
I20240501 19:56:45.345396 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.342519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:54611 (local address 127.8.168.67:34613)
0501 19:56:45.342701 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:45.342706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:45.342795 (+ 89us) server_negotiation.cc:407] Connection header received
0501 19:56:45.342922 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:45.342924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:45.342969 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:45.343042 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:45.343392 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.343856 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.344680 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.345146 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.345183 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:45.345235 (+ 52us) server_negotiation.cc:299] Negotiation successful
0501 19:56:45.345286 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20240501 19:56:45.345464 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.342450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:37655 (local address 127.8.168.65:45667)
0501 19:56:45.342682 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:45.342686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:45.342704 (+ 18us) server_negotiation.cc:407] Connection header received
0501 19:56:45.342853 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:45.342857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:45.342908 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:45.342974 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:45.343373 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.343852 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.344672 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.345211 (+ 539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.345250 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:45.345303 (+ 53us) server_negotiation.cc:299] Negotiation successful
0501 19:56:45.345357 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20240501 19:56:45.345789 16850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eee4f3079284eabbeba69d24e500dcf" candidate_uuid: "e914b9526b3b43bda46fb11aeb9324b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56d95af6390d40e7a395ff0a2d798726" is_pre_election: true
I20240501 19:56:45.345927 16850 raft_consensus.cc:2454] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e914b9526b3b43bda46fb11aeb9324b7 in term 0.
I20240501 19:56:45.345902 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eee4f3079284eabbeba69d24e500dcf" candidate_uuid: "e914b9526b3b43bda46fb11aeb9324b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f30e604459744605bbdeb2fe62c9396b" is_pre_election: true
I20240501 19:56:45.346114 16570 raft_consensus.cc:2454] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e914b9526b3b43bda46fb11aeb9324b7 in term 0.
I20240501 19:56:45.346269 16641 leader_election.cc:302] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56d95af6390d40e7a395ff0a2d798726, e914b9526b3b43bda46fb11aeb9324b7; no voters:
I20240501 19:56:45.346486 16932 raft_consensus.cc:2781] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:45.346541 16932 raft_consensus.cc:482] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:45.346586 16932 raft_consensus.cc:3037] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.347433 16932 raft_consensus.cc:504] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.347745 16932 leader_election.cc:288] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [CANDIDATE]: Term 1 election: Requested vote from peers f30e604459744605bbdeb2fe62c9396b (127.8.168.65:45667), 56d95af6390d40e7a395ff0a2d798726 (127.8.168.67:34613)
I20240501 19:56:45.348094 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eee4f3079284eabbeba69d24e500dcf" candidate_uuid: "e914b9526b3b43bda46fb11aeb9324b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f30e604459744605bbdeb2fe62c9396b"
I20240501 19:56:45.348114 16850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eee4f3079284eabbeba69d24e500dcf" candidate_uuid: "e914b9526b3b43bda46fb11aeb9324b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56d95af6390d40e7a395ff0a2d798726"
I20240501 19:56:45.348213 16570 raft_consensus.cc:3037] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.348213 16850 raft_consensus.cc:3037] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.349236 16850 raft_consensus.cc:2454] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e914b9526b3b43bda46fb11aeb9324b7 in term 1.
I20240501 19:56:45.349236 16570 raft_consensus.cc:2454] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e914b9526b3b43bda46fb11aeb9324b7 in term 1.
I20240501 19:56:45.349521 16642 leader_election.cc:302] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e914b9526b3b43bda46fb11aeb9324b7, f30e604459744605bbdeb2fe62c9396b; no voters:
I20240501 19:56:45.349681 16932 raft_consensus.cc:2781] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:45.349882 16932 raft_consensus.cc:686] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [term 1 LEADER]: Becoming Leader. State: Replica: e914b9526b3b43bda46fb11aeb9324b7, State: Running, Role: LEADER
I20240501 19:56:45.350122 16932 consensus_queue.cc:237] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.351207 16965 tablet_replica.cc:439] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7: TxnStatusTablet state changed. Reason: New leader e914b9526b3b43bda46fb11aeb9324b7. Latest consensus state: current_term: 1 leader_uuid: "e914b9526b3b43bda46fb11aeb9324b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } }
I20240501 19:56:45.351308 16965 tablet_replica.cc:442] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7: This TxnStatusTablet replica's current role is: LEADER
I20240501 19:56:45.351540 16967 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240501 19:56:45.351648 16967 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240501 19:56:45.352310 16412 catalog_manager.cc:5484] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e914b9526b3b43bda46fb11aeb9324b7 (127.8.168.66). New cstate: current_term: 1 leader_uuid: "e914b9526b3b43bda46fb11aeb9324b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:45.405944 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.402551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:34269 (local address 127.0.0.1:42928)
0501 19:56:45.402731 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:45.402746 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:45.402849 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:45.403213 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:45.403216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:45.403224 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 19:56:45.403474 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.403481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.404130 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.404134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:45.404818 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.404828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.404937 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.404991 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 19:56:45.405655 (+ 664us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 19:56:45.405668 (+ 13us) client_negotiation.cc:769] Sending connection context
0501 19:56:45.405732 (+ 64us) client_negotiation.cc:240] Negotiation successful
0501 19:56:45.405803 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20240501 19:56:45.405931 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.402689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42928 (local address 127.8.168.66:34269)
0501 19:56:45.402812 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:45.402815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:45.402829 (+ 14us) server_negotiation.cc:407] Connection header received
0501 19:56:45.403038 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:45.403041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:45.403095 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:45.403181 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:45.403564 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.404039 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.404930 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.405088 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.405144 (+ 56us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:45.405550 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:45.405662 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:45.405745 (+ 83us) server_negotiation.cc:299] Negotiation successful
0501 19:56:45.405791 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240501 19:56:45.408768 16850 raft_consensus.cc:1261] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726 [term 1 FOLLOWER]: Refusing update from remote peer e914b9526b3b43bda46fb11aeb9324b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.408885 16570 raft_consensus.cc:1261] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b [term 1 FOLLOWER]: Refusing update from remote peer e914b9526b3b43bda46fb11aeb9324b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.409380 16932 consensus_queue.cc:1035] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.409546 16965 consensus_queue.cc:1035] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.411206 16931 tablet_replica.cc:439] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726: TxnStatusTablet state changed. Reason: New leader e914b9526b3b43bda46fb11aeb9324b7. Latest consensus state: current_term: 1 leader_uuid: "e914b9526b3b43bda46fb11aeb9324b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } }
I20240501 19:56:45.411314 16931 tablet_replica.cc:442] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:45.411468 16930 tablet_replica.cc:439] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b: TxnStatusTablet state changed. Reason: New leader e914b9526b3b43bda46fb11aeb9324b7. Latest consensus state: current_term: 1 leader_uuid: "e914b9526b3b43bda46fb11aeb9324b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } }
I20240501 19:56:45.411566 16930 tablet_replica.cc:442] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:45.413447 16932 tablet_replica.cc:439] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e914b9526b3b43bda46fb11aeb9324b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } }
I20240501 19:56:45.413535 16932 tablet_replica.cc:442] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7: This TxnStatusTablet replica's current role is: LEADER
I20240501 19:56:45.413417 16965 tablet_replica.cc:439] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e914b9526b3b43bda46fb11aeb9324b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } }
I20240501 19:56:45.413717 16965 tablet_replica.cc:442] T 2eee4f3079284eabbeba69d24e500dcf P e914b9526b3b43bda46fb11aeb9324b7: This TxnStatusTablet replica's current role is: LEADER
I20240501 19:56:45.415674 16931 tablet_replica.cc:439] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e914b9526b3b43bda46fb11aeb9324b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } }
I20240501 19:56:45.415793 16931 tablet_replica.cc:442] T 2eee4f3079284eabbeba69d24e500dcf P 56d95af6390d40e7a395ff0a2d798726: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:45.416777 16930 tablet_replica.cc:439] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e914b9526b3b43bda46fb11aeb9324b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } } }
I20240501 19:56:45.416882 16930 tablet_replica.cc:442] T 2eee4f3079284eabbeba69d24e500dcf P f30e604459744605bbdeb2fe62c9396b: This TxnStatusTablet replica's current role is: FOLLOWER
I20240501 19:56:45.428274 16975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.425229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:34269 (local address 127.8.168.65:33181)
0501 19:56:45.425529 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:45.425542 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:45.425659 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:45.425970 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:45.425974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:45.425982 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 19:56:45.426193 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.426200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.426892 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.426896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:45.427315 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.427322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.427443 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.427496 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 19:56:45.427843 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 19:56:45.427851 (+ 8us) client_negotiation.cc:769] Sending connection context
0501 19:56:45.428058 (+ 207us) client_negotiation.cc:240] Negotiation successful
0501 19:56:45.428144 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":84,"threads_started":1}
I20240501 19:56:45.428272 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.425342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:33181 (local address 127.8.168.66:34269)
0501 19:56:45.425481 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:45.425486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:45.425613 (+ 127us) server_negotiation.cc:407] Connection header received
0501 19:56:45.425778 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:45.425780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:45.425831 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:45.425949 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:45.426301 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.426780 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.427432 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.427621 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.427678 (+ 57us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:45.427754 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:45.427826 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:45.428080 (+ 254us) server_negotiation.cc:299] Negotiation successful
0501 19:56:45.428129 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20240501 19:56:45.435083 16850 raft_consensus.cc:1261] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726 [term 1 FOLLOWER]: Refusing update from remote peer f30e604459744605bbdeb2fe62c9396b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.435178 16710 raft_consensus.cc:1261] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7 [term 1 FOLLOWER]: Refusing update from remote peer f30e604459744605bbdeb2fe62c9396b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.435547 16935 consensus_queue.cc:1035] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [LEADER]: Connected to new peer: Peer: permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.435756 16930 consensus_queue.cc:1035] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.458097 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.454736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:45667 (local address 127.8.168.66:50687)
0501 19:56:45.455089 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:45.455099 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:45.455196 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:45.455409 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:45.455412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:45.455420 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 19:56:45.456020 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.456029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.456921 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.456925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:45.457340 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.457348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.457442 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.457480 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 19:56:45.457858 (+ 378us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 19:56:45.457862 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 19:56:45.457906 (+ 44us) client_negotiation.cc:240] Negotiation successful
0501 19:56:45.457973 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":96,"threads_started":1}
I20240501 19:56:45.458099 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.454799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:50687 (local address 127.8.168.65:45667)
0501 19:56:45.454917 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:45.454921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:45.455165 (+ 244us) server_negotiation.cc:407] Connection header received
0501 19:56:45.455287 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:45.455290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:45.455330 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:45.455388 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:45.456170 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.456804 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.457441 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.457651 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.457698 (+ 47us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:45.457779 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:45.457853 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:45.457929 (+ 76us) server_negotiation.cc:299] Negotiation successful
0501 19:56:45.457977 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20240501 19:56:45.482916 16411 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51980:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:45.483103 16411 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51980:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:45.485529 16411 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417: Sending DeleteTablet for 3 replicas of tablet e08ba3cfd7ba408f8bd6caf4891db9a2
I20240501 19:56:45.486299 16830 tablet_service.cc:1514] Processing DeleteTablet for tablet e08ba3cfd7ba408f8bd6caf4891db9a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='oryx', principal='oryx/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:42818
I20240501 19:56:45.486526 16948 tablet_replica.cc:330] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726: stopping tablet replica
I20240501 19:56:45.486515 16550 tablet_service.cc:1514] Processing DeleteTablet for tablet e08ba3cfd7ba408f8bd6caf4891db9a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='oryx', principal='oryx/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:43730
I20240501 19:56:45.486640 16948 raft_consensus.cc:2229] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.486775 16948 raft_consensus.cc:2258] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.486799 16949 tablet_replica.cc:330] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b: stopping tablet replica
I20240501 19:56:45.486891 16949 raft_consensus.cc:2229] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:45.487048 16949 raft_consensus.cc:2258] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.487709 16949 ts_tablet_manager.cc:1875] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.489423 16948 ts_tablet_manager.cc:1875] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.490374 16949 ts_tablet_manager.cc:1888] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 19:56:45.490444 16949 log.cc:1196] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/wal/wals/e08ba3cfd7ba408f8bd6caf4891db9a2
I20240501 19:56:45.490649 16949 ts_tablet_manager.cc:1909] T e08ba3cfd7ba408f8bd6caf4891db9a2 P f30e604459744605bbdeb2fe62c9396b: Deleting consensus metadata
I20240501 19:56:45.491302 16398 catalog_manager.cc:4830] TS f30e604459744605bbdeb2fe62c9396b (127.8.168.65:45667): tablet e08ba3cfd7ba408f8bd6caf4891db9a2 (table test-table [id=78557681920c4e4b8bfedb056b01fdec]) successfully deleted
I20240501 19:56:45.492043 16948 ts_tablet_manager.cc:1888] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 19:56:45.492269 16688 tablet_service.cc:1514] Processing DeleteTablet for tablet e08ba3cfd7ba408f8bd6caf4891db9a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='oryx', principal='oryx/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:42920
I20240501 19:56:45.492462 16947 tablet_replica.cc:330] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7: stopping tablet replica
I20240501 19:56:45.492568 16947 raft_consensus.cc:2229] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.492689 16947 raft_consensus.cc:2258] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.493379 16947 ts_tablet_manager.cc:1875] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.495227 16947 ts_tablet_manager.cc:1888] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 19:56:45.495285 16947 log.cc:1196] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/e08ba3cfd7ba408f8bd6caf4891db9a2
I20240501 19:56:45.495451 16947 ts_tablet_manager.cc:1909] T e08ba3cfd7ba408f8bd6caf4891db9a2 P e914b9526b3b43bda46fb11aeb9324b7: Deleting consensus metadata
I20240501 19:56:45.496116 16397 catalog_manager.cc:4830] TS e914b9526b3b43bda46fb11aeb9324b7 (127.8.168.66:34269): tablet e08ba3cfd7ba408f8bd6caf4891db9a2 (table test-table [id=78557681920c4e4b8bfedb056b01fdec]) successfully deleted
I20240501 19:56:45.497016 16948 log.cc:1196] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/e08ba3cfd7ba408f8bd6caf4891db9a2
I20240501 19:56:45.497318 16948 ts_tablet_manager.cc:1909] T e08ba3cfd7ba408f8bd6caf4891db9a2 P 56d95af6390d40e7a395ff0a2d798726: Deleting consensus metadata
I20240501 19:56:45.498080 16397 catalog_manager.cc:4830] TS 56d95af6390d40e7a395ff0a2d798726 (127.8.168.67:34613): tablet e08ba3cfd7ba408f8bd6caf4891db9a2 (table test-table [id=78557681920c4e4b8bfedb056b01fdec]) successfully deleted
May 01 19:56:45 dist-test-slave-tzct krb5kdc[16368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 01 19:56:45 dist-test-slave-tzct krb5kdc[16368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:45 dist-test-slave-tzct krb5kdc[16368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
W20240501 19:56:45.517038 16462 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:45.511536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51992 (local address 127.8.168.126:35023)
0501 19:56:45.511665 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:45.511670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:45.511869 (+ 199us) server_negotiation.cc:407] Connection header received
0501 19:56:45.511997 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:45.511999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:45.512047 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:45.512116 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:45.513041 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.513528 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.514166 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.514346 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.516903 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51992: BlockingRecv error: recv got EOF from 127.0.0.1:51992 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
May 01 19:56:45 dist-test-slave-tzct krb5kdc[16368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM
I20240501 19:56:45.526325 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.518806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.8.168.126:35023)
0501 19:56:45.518941 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:45.518945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:45.518968 (+ 23us) server_negotiation.cc:407] Connection header received
0501 19:56:45.519091 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:45.519094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:45.519136 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:45.519203 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:45.519858 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.520332 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.520961 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.521140 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.523304 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:45.523321 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:45.523323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:45.523347 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:45.524831 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:45.525250 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:45.525253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:45.525254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:45.525295 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:45.525516 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:45.525519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:45.525520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:45.525661 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:45.525762 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:45.526066 (+ 304us) server_negotiation.cc:299] Negotiation successful
0501 19:56:45.526163 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20240501 19:56:45.529057 16411 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:56:45.529503 16411 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:45.534301 16550 tablet_service.cc:1467] Processing CreateTablet for tablet 8528cc72af974b4c9a956bc9f20eb140 (DEFAULT_TABLE table=test-table [id=5f8b42580aa346929d8d75533a644bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.534675 16550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8528cc72af974b4c9a956bc9f20eb140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.534634 16830 tablet_service.cc:1467] Processing CreateTablet for tablet 8528cc72af974b4c9a956bc9f20eb140 (DEFAULT_TABLE table=test-table [id=5f8b42580aa346929d8d75533a644bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.534875 16830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8528cc72af974b4c9a956bc9f20eb140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.536979 16926 tablet_bootstrap.cc:492] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726: Bootstrap starting.
I20240501 19:56:45.536979 16924 tablet_bootstrap.cc:492] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b: Bootstrap starting.
I20240501 19:56:45.537047 16688 tablet_service.cc:1467] Processing CreateTablet for tablet 8528cc72af974b4c9a956bc9f20eb140 (DEFAULT_TABLE table=test-table [id=5f8b42580aa346929d8d75533a644bf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.537322 16688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8528cc72af974b4c9a956bc9f20eb140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.537847 16926 tablet_bootstrap.cc:654] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.537847 16924 tablet_bootstrap.cc:654] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.538967 16926 tablet_bootstrap.cc:492] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726: No bootstrap required, opened a new log
I20240501 19:56:45.539058 16926 ts_tablet_manager.cc:1367] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:45.539067 16924 tablet_bootstrap.cc:492] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b: No bootstrap required, opened a new log
I20240501 19:56:45.539134 16924 ts_tablet_manager.cc:1367] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:45.539383 16925 tablet_bootstrap.cc:492] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7: Bootstrap starting.
I20240501 19:56:45.539508 16924 raft_consensus.cc:348] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.539508 16926 raft_consensus.cc:348] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.539606 16924 raft_consensus.cc:374] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.539606 16926 raft_consensus.cc:374] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.539636 16924 raft_consensus.cc:729] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f30e604459744605bbdeb2fe62c9396b, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.539636 16926 raft_consensus.cc:729] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d95af6390d40e7a395ff0a2d798726, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.539757 16926 consensus_queue.cc:260] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.539757 16924 consensus_queue.cc:260] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.540010 16924 ts_tablet_manager.cc:1398] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:45.540006 16926 ts_tablet_manager.cc:1398] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:45.540130 16925 tablet_bootstrap.cc:654] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.541172 16925 tablet_bootstrap.cc:492] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7: No bootstrap required, opened a new log
I20240501 19:56:45.541261 16925 ts_tablet_manager.cc:1367] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:45.541657 16925 raft_consensus.cc:348] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.541764 16925 raft_consensus.cc:374] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.541798 16925 raft_consensus.cc:729] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e914b9526b3b43bda46fb11aeb9324b7, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.541913 16925 consensus_queue.cc:260] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.542156 16925 ts_tablet_manager.cc:1398] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:56:45.682091 16972 raft_consensus.cc:482] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.682227 16972 raft_consensus.cc:504] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.682627 16972 leader_election.cc:288] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f30e604459744605bbdeb2fe62c9396b (127.8.168.65:45667), 56d95af6390d40e7a395ff0a2d798726 (127.8.168.67:34613)
I20240501 19:56:45.683108 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8528cc72af974b4c9a956bc9f20eb140" candidate_uuid: "e914b9526b3b43bda46fb11aeb9324b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f30e604459744605bbdeb2fe62c9396b" is_pre_election: true
I20240501 19:56:45.683121 16850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8528cc72af974b4c9a956bc9f20eb140" candidate_uuid: "e914b9526b3b43bda46fb11aeb9324b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56d95af6390d40e7a395ff0a2d798726" is_pre_election: true
I20240501 19:56:45.683243 16850 raft_consensus.cc:2454] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e914b9526b3b43bda46fb11aeb9324b7 in term 0.
I20240501 19:56:45.683256 16570 raft_consensus.cc:2454] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e914b9526b3b43bda46fb11aeb9324b7 in term 0.
I20240501 19:56:45.683538 16642 leader_election.cc:302] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e914b9526b3b43bda46fb11aeb9324b7, f30e604459744605bbdeb2fe62c9396b; no voters:
I20240501 19:56:45.683737 16972 raft_consensus.cc:2781] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:45.683791 16972 raft_consensus.cc:482] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:45.683818 16972 raft_consensus.cc:3037] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.684711 16972 raft_consensus.cc:504] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.685071 16972 leader_election.cc:288] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [CANDIDATE]: Term 1 election: Requested vote from peers f30e604459744605bbdeb2fe62c9396b (127.8.168.65:45667), 56d95af6390d40e7a395ff0a2d798726 (127.8.168.67:34613)
I20240501 19:56:45.685443 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8528cc72af974b4c9a956bc9f20eb140" candidate_uuid: "e914b9526b3b43bda46fb11aeb9324b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f30e604459744605bbdeb2fe62c9396b"
I20240501 19:56:45.685465 16850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8528cc72af974b4c9a956bc9f20eb140" candidate_uuid: "e914b9526b3b43bda46fb11aeb9324b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56d95af6390d40e7a395ff0a2d798726"
I20240501 19:56:45.685544 16570 raft_consensus.cc:3037] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.685590 16850 raft_consensus.cc:3037] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.686352 16570 raft_consensus.cc:2454] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e914b9526b3b43bda46fb11aeb9324b7 in term 1.
I20240501 19:56:45.686507 16850 raft_consensus.cc:2454] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e914b9526b3b43bda46fb11aeb9324b7 in term 1.
I20240501 19:56:45.686631 16642 leader_election.cc:302] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e914b9526b3b43bda46fb11aeb9324b7, f30e604459744605bbdeb2fe62c9396b; no voters:
I20240501 19:56:45.686808 16972 raft_consensus.cc:2781] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:45.686883 16972 raft_consensus.cc:686] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [term 1 LEADER]: Becoming Leader. State: Replica: e914b9526b3b43bda46fb11aeb9324b7, State: Running, Role: LEADER
I20240501 19:56:45.687027 16972 consensus_queue.cc:237] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.688810 16412 catalog_manager.cc:5484] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e914b9526b3b43bda46fb11aeb9324b7 (127.8.168.66). New cstate: current_term: 1 leader_uuid: "e914b9526b3b43bda46fb11aeb9324b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:45.730003 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.727572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.8.168.66:34269)
0501 19:56:45.727712 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:45.727717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:45.727732 (+ 15us) server_negotiation.cc:407] Connection header received
0501 19:56:45.727783 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:45.727787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:45.727833 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:45.727894 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:45.728233 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.728685 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.729356 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.729524 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.729566 (+ 42us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:45.729632 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:45.729694 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:45.729812 (+ 118us) server_negotiation.cc:299] Negotiation successful
0501 19:56:45.729857 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20240501 19:56:45.732064 16850 raft_consensus.cc:1261] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726 [term 1 FOLLOWER]: Refusing update from remote peer e914b9526b3b43bda46fb11aeb9324b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.732091 16570 raft_consensus.cc:1261] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b [term 1 FOLLOWER]: Refusing update from remote peer e914b9526b3b43bda46fb11aeb9324b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.732482 16932 consensus_queue.cc:1035] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.732597 16972 consensus_queue.cc:1035] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.736868 16940 mvcc.cc:205] Tried to move back new op lower bound from 7022974589874741248 to 7022974589695619072. Current Snapshot: MvccSnapshot[applied={T|T < 7022974589874741248 or (T in {7022974589874741248})}]
I20240501 19:56:45.741547 16411 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52006:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:45.741672 16411 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52006:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:45.743527 16411 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417: Sending DeleteTablet for 3 replicas of tablet 8528cc72af974b4c9a956bc9f20eb140
I20240501 19:56:45.744067 16688 tablet_service.cc:1514] Processing DeleteTablet for tablet 8528cc72af974b4c9a956bc9f20eb140 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='oryx', principal='oryx/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:42920
I20240501 19:56:45.744267 16550 tablet_service.cc:1514] Processing DeleteTablet for tablet 8528cc72af974b4c9a956bc9f20eb140 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='oryx', principal='oryx/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:43730
I20240501 19:56:45.744313 16947 tablet_replica.cc:330] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7: stopping tablet replica
I20240501 19:56:45.744318 16830 tablet_service.cc:1514] Processing DeleteTablet for tablet 8528cc72af974b4c9a956bc9f20eb140 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:45 UTC) from {username='oryx', principal='oryx/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:42818
I20240501 19:56:45.744417 16947 raft_consensus.cc:2229] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:45.744421 16949 tablet_replica.cc:330] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b: stopping tablet replica
I20240501 19:56:45.744489 16948 tablet_replica.cc:330] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726: stopping tablet replica
I20240501 19:56:45.744524 16949 raft_consensus.cc:2229] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.744587 16948 raft_consensus.cc:2229] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:45.744632 16949 raft_consensus.cc:2258] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.744697 16948 raft_consensus.cc:2258] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.744696 16947 raft_consensus.cc:2258] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:45.745410 16949 ts_tablet_manager.cc:1875] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.745537 16947 ts_tablet_manager.cc:1875] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.745545 16948 ts_tablet_manager.cc:1875] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:45.747609 16947 ts_tablet_manager.cc:1888] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:45.747609 16949 ts_tablet_manager.cc:1888] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:45.747674 16949 log.cc:1196] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/wal/wals/8528cc72af974b4c9a956bc9f20eb140
I20240501 19:56:45.747674 16947 log.cc:1196] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/8528cc72af974b4c9a956bc9f20eb140
I20240501 19:56:45.747691 16948 ts_tablet_manager.cc:1888] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:56:45.747756 16948 log.cc:1196] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/8528cc72af974b4c9a956bc9f20eb140
I20240501 19:56:45.747905 16949 ts_tablet_manager.cc:1909] T 8528cc72af974b4c9a956bc9f20eb140 P f30e604459744605bbdeb2fe62c9396b: Deleting consensus metadata
I20240501 19:56:45.747905 16947 ts_tablet_manager.cc:1909] T 8528cc72af974b4c9a956bc9f20eb140 P e914b9526b3b43bda46fb11aeb9324b7: Deleting consensus metadata
I20240501 19:56:45.747989 16948 ts_tablet_manager.cc:1909] T 8528cc72af974b4c9a956bc9f20eb140 P 56d95af6390d40e7a395ff0a2d798726: Deleting consensus metadata
I20240501 19:56:45.748627 16398 catalog_manager.cc:4830] TS f30e604459744605bbdeb2fe62c9396b (127.8.168.65:45667): tablet 8528cc72af974b4c9a956bc9f20eb140 (table test-table [id=5f8b42580aa346929d8d75533a644bf4]) successfully deleted
I20240501 19:56:45.748826 16397 catalog_manager.cc:4830] TS e914b9526b3b43bda46fb11aeb9324b7 (127.8.168.66:34269): tablet 8528cc72af974b4c9a956bc9f20eb140 (table test-table [id=5f8b42580aa346929d8d75533a644bf4]) successfully deleted
I20240501 19:56:45.748968 16397 catalog_manager.cc:4830] TS 56d95af6390d40e7a395ff0a2d798726 (127.8.168.67:34613): tablet 8528cc72af974b4c9a956bc9f20eb140 (table test-table [id=5f8b42580aa346929d8d75533a644bf4]) successfully deleted
I20240501 19:56:45.749825 16411 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:56:45.750223 16411 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:45.754835 16550 tablet_service.cc:1467] Processing CreateTablet for tablet 9be119e1fbe74e5f84fa96d3eddadeef (DEFAULT_TABLE table=test-table [id=d87d3345cdf24f7b9b3f72c854ecdf5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.755160 16550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be119e1fbe74e5f84fa96d3eddadeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.755415 16830 tablet_service.cc:1467] Processing CreateTablet for tablet 9be119e1fbe74e5f84fa96d3eddadeef (DEFAULT_TABLE table=test-table [id=d87d3345cdf24f7b9b3f72c854ecdf5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.755673 16830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be119e1fbe74e5f84fa96d3eddadeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.757009 16688 tablet_service.cc:1467] Processing CreateTablet for tablet 9be119e1fbe74e5f84fa96d3eddadeef (DEFAULT_TABLE table=test-table [id=d87d3345cdf24f7b9b3f72c854ecdf5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.757277 16688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be119e1fbe74e5f84fa96d3eddadeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.757526 16924 tablet_bootstrap.cc:492] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b: Bootstrap starting.
I20240501 19:56:45.758209 16924 tablet_bootstrap.cc:654] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.758646 16926 tablet_bootstrap.cc:492] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726: Bootstrap starting.
I20240501 19:56:45.759413 16926 tablet_bootstrap.cc:654] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.759575 16925 tablet_bootstrap.cc:492] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7: Bootstrap starting.
I20240501 19:56:45.760507 16924 tablet_bootstrap.cc:492] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b: No bootstrap required, opened a new log
I20240501 19:56:45.760560 16925 tablet_bootstrap.cc:654] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.760592 16924 ts_tablet_manager.cc:1367] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:45.760982 16924 raft_consensus.cc:348] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.761117 16924 raft_consensus.cc:374] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.761155 16924 raft_consensus.cc:729] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f30e604459744605bbdeb2fe62c9396b, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.761276 16924 consensus_queue.cc:260] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.761559 16924 ts_tablet_manager.cc:1398] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:45.761837 16925 tablet_bootstrap.cc:492] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7: No bootstrap required, opened a new log
I20240501 19:56:45.761919 16925 ts_tablet_manager.cc:1367] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:45.762313 16925 raft_consensus.cc:348] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.762415 16925 raft_consensus.cc:374] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.762450 16925 raft_consensus.cc:729] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e914b9526b3b43bda46fb11aeb9324b7, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.762558 16925 consensus_queue.cc:260] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.762614 16926 tablet_bootstrap.cc:492] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726: No bootstrap required, opened a new log
I20240501 19:56:45.762689 16926 ts_tablet_manager.cc:1367] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:45.763075 16926 raft_consensus.cc:348] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.763162 16926 raft_consensus.cc:374] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.763191 16926 raft_consensus.cc:729] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d95af6390d40e7a395ff0a2d798726, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.763309 16926 consensus_queue.cc:260] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.763571 16926 ts_tablet_manager.cc:1398] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:45.765584 16925 ts_tablet_manager.cc:1398] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:45.932148 16931 raft_consensus.cc:482] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.932286 16931 raft_consensus.cc:504] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.933320 16931 leader_election.cc:288] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f30e604459744605bbdeb2fe62c9396b (127.8.168.65:45667), e914b9526b3b43bda46fb11aeb9324b7 (127.8.168.66:34269)
I20240501 19:56:45.936152 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.933621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:34269 (local address 127.8.168.67:51299)
0501 19:56:45.933935 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:45.933946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:45.934047 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:45.934324 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:45.934327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:45.934345 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:45.934536 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.934542 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.935199 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.935203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:45.935773 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.935781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.935874 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.935892 (+ 18us) client_negotiation.cc:769] Sending connection context
0501 19:56:45.935937 (+ 45us) client_negotiation.cc:240] Negotiation successful
0501 19:56:45.935992 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":64,"threads_started":1}
I20240501 19:56:45.936300 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.933586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:45667 (local address 127.8.168.67:52727)
0501 19:56:45.933886 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:45.933897 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:45.934009 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:45.934272 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:45.934275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:45.934292 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:56:45.934509 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.934515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.935242 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.935244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:45.935829 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.935838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.935968 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.935985 (+ 17us) client_negotiation.cc:769] Sending connection context
0501 19:56:45.936034 (+ 49us) client_negotiation.cc:240] Negotiation successful
0501 19:56:45.936088 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":83,"threads_started":1}
I20240501 19:56:45.936579 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.933680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:51299 (local address 127.8.168.66:34269)
0501 19:56:45.933803 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:45.933808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:45.934023 (+ 215us) server_negotiation.cc:407] Connection header received
0501 19:56:45.934152 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:45.934155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:45.934195 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:45.934259 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:45.934645 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.935101 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.935884 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.936336 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.936375 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:45.936426 (+ 51us) server_negotiation.cc:299] Negotiation successful
0501 19:56:45.936471 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 19:56:45.936664 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.933654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:52727 (local address 127.8.168.65:45667)
0501 19:56:45.933803 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:45.933808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:45.933972 (+ 164us) server_negotiation.cc:407] Connection header received
0501 19:56:45.934109 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:45.934111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:45.934160 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:45.934241 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:56:45.934617 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.935131 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.935954 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.936398 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.936437 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:45.936489 (+ 52us) server_negotiation.cc:299] Negotiation successful
0501 19:56:45.936543 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20240501 19:56:45.936975 16710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be119e1fbe74e5f84fa96d3eddadeef" candidate_uuid: "56d95af6390d40e7a395ff0a2d798726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e914b9526b3b43bda46fb11aeb9324b7" is_pre_election: true
I20240501 19:56:45.937053 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be119e1fbe74e5f84fa96d3eddadeef" candidate_uuid: "56d95af6390d40e7a395ff0a2d798726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f30e604459744605bbdeb2fe62c9396b" is_pre_election: true
I20240501 19:56:45.937124 16710 raft_consensus.cc:2454] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56d95af6390d40e7a395ff0a2d798726 in term 0.
I20240501 19:56:45.937196 16570 raft_consensus.cc:2454] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56d95af6390d40e7a395ff0a2d798726 in term 0.
I20240501 19:56:45.937487 16781 leader_election.cc:302] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 56d95af6390d40e7a395ff0a2d798726, e914b9526b3b43bda46fb11aeb9324b7, f30e604459744605bbdeb2fe62c9396b; no voters:
I20240501 19:56:45.937757 16931 raft_consensus.cc:2781] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:45.937811 16931 raft_consensus.cc:482] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:45.937847 16931 raft_consensus.cc:3037] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.938666 16931 raft_consensus.cc:504] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.938997 16931 leader_election.cc:288] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [CANDIDATE]: Term 1 election: Requested vote from peers f30e604459744605bbdeb2fe62c9396b (127.8.168.65:45667), e914b9526b3b43bda46fb11aeb9324b7 (127.8.168.66:34269)
I20240501 19:56:45.939297 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be119e1fbe74e5f84fa96d3eddadeef" candidate_uuid: "56d95af6390d40e7a395ff0a2d798726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f30e604459744605bbdeb2fe62c9396b"
I20240501 19:56:45.939306 16710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be119e1fbe74e5f84fa96d3eddadeef" candidate_uuid: "56d95af6390d40e7a395ff0a2d798726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e914b9526b3b43bda46fb11aeb9324b7"
I20240501 19:56:45.939409 16570 raft_consensus.cc:3037] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.939412 16710 raft_consensus.cc:3037] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.940152 16710 raft_consensus.cc:2454] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56d95af6390d40e7a395ff0a2d798726 in term 1.
I20240501 19:56:45.940178 16570 raft_consensus.cc:2454] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56d95af6390d40e7a395ff0a2d798726 in term 1.
I20240501 19:56:45.940392 16781 leader_election.cc:302] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56d95af6390d40e7a395ff0a2d798726, e914b9526b3b43bda46fb11aeb9324b7; no voters:
I20240501 19:56:45.940564 16931 raft_consensus.cc:2781] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:45.940734 16931 raft_consensus.cc:686] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [term 1 LEADER]: Becoming Leader. State: Replica: 56d95af6390d40e7a395ff0a2d798726, State: Running, Role: LEADER
I20240501 19:56:45.940966 16931 consensus_queue.cc:237] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } }
I20240501 19:56:45.942937 16413 catalog_manager.cc:5484] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56d95af6390d40e7a395ff0a2d798726 (127.8.168.67). New cstate: current_term: 1 leader_uuid: "56d95af6390d40e7a395ff0a2d798726" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56d95af6390d40e7a395ff0a2d798726" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34613 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:45.961761 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.958815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.8.168.67:34613)
0501 19:56:45.959032 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:45.959034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:45.959045 (+ 11us) server_negotiation.cc:407] Connection header received
0501 19:56:45.959110 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:45.959112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:45.959160 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:45.959264 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:45.959512 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.960089 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.960699 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.960871 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.960928 (+ 57us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:45.961416 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:45.961499 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:45.961578 (+ 79us) server_negotiation.cc:299] Negotiation successful
0501 19:56:45.961621 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":86,"threads_started":1}
I20240501 19:56:45.970530 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.967850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:34269 (local address 127.8.168.67:59943)
0501 19:56:45.968124 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:45.968136 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:45.968230 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:45.968455 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:45.968458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:45.968465 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 19:56:45.968706 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.968712 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.969374 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.969378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:45.969793 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.969800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.969896 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.969939 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 19:56:45.970257 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 19:56:45.970264 (+ 7us) client_negotiation.cc:769] Sending connection context
0501 19:56:45.970320 (+ 56us) client_negotiation.cc:240] Negotiation successful
0501 19:56:45.970388 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
I20240501 19:56:45.970510 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.967943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:59943 (local address 127.8.168.66:34269)
0501 19:56:45.968073 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:45.968077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:45.968195 (+ 118us) server_negotiation.cc:407] Connection header received
0501 19:56:45.968329 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:45.968332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:45.968377 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:45.968438 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:45.968822 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.969278 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.969893 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.970065 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.970114 (+ 49us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:45.970177 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:45.970236 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:45.970337 (+ 101us) server_negotiation.cc:299] Negotiation successful
0501 19:56:45.970376 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20240501 19:56:45.975700 16710 raft_consensus.cc:1261] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7 [term 1 FOLLOWER]: Refusing update from remote peer 56d95af6390d40e7a395ff0a2d798726: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.975700 16570 raft_consensus.cc:1261] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b [term 1 FOLLOWER]: Refusing update from remote peer 56d95af6390d40e7a395ff0a2d798726: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.976182 17007 consensus_queue.cc:1035] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f30e604459744605bbdeb2fe62c9396b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.976359 16931 consensus_queue.cc:1035] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e914b9526b3b43bda46fb11aeb9324b7" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 34269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.991771 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.989188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:34613 (local address 127.8.168.66:38405)
0501 19:56:45.989425 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:45.989439 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:45.989524 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:45.989740 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:45.989742 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:45.989747 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0501 19:56:45.989945 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.989951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.990688 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.990691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:45.991110 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:45.991118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.991203 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.991227 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0501 19:56:45.991493 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0501 19:56:45.991497 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 19:56:45.991542 (+ 45us) client_negotiation.cc:240] Negotiation successful
0501 19:56:45.991629 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20240501 19:56:45.991861 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:45.989469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:38405 (local address 127.8.168.67:34613)
0501 19:56:45.989582 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:45.989585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:45.989595 (+ 10us) server_negotiation.cc:407] Connection header received
0501 19:56:45.989629 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:45.989631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:45.989670 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:45.989730 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:56:45.990080 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.990603 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:45.991195 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:45.991331 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:45.991367 (+ 36us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:56:45.991430 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:56:45.991485 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:45.991672 (+ 187us) server_negotiation.cc:299] Negotiation successful
0501 19:56:45.991733 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240501 19:56:46.015231 16411 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52006:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:46.015384 16411 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52006:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:56:46.017490 16411 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 6fa2c4ee823246bca0fa71d967a21417: Sending DeleteTablet for 3 replicas of tablet 9be119e1fbe74e5f84fa96d3eddadeef
I20240501 19:56:46.018059 16550 tablet_service.cc:1514] Processing DeleteTablet for tablet 9be119e1fbe74e5f84fa96d3eddadeef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='oryx', principal='oryx/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:43730
I20240501 19:56:46.018168 16830 tablet_service.cc:1514] Processing DeleteTablet for tablet 9be119e1fbe74e5f84fa96d3eddadeef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='oryx', principal='oryx/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:42818
I20240501 19:56:46.018167 16688 tablet_service.cc:1514] Processing DeleteTablet for tablet 9be119e1fbe74e5f84fa96d3eddadeef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:56:46 UTC) from {username='oryx', principal='oryx/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:42920
I20240501 19:56:46.018278 16949 tablet_replica.cc:330] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b: stopping tablet replica
I20240501 19:56:46.018328 16948 tablet_replica.cc:330] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726: stopping tablet replica
I20240501 19:56:46.018329 16947 tablet_replica.cc:330] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7: stopping tablet replica
I20240501 19:56:46.018401 16949 raft_consensus.cc:2229] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.018422 16948 raft_consensus.cc:2229] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:46.018440 16947 raft_consensus.cc:2229] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.018512 16949 raft_consensus.cc:2258] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.018533 16947 raft_consensus.cc:2258] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.018705 16948 raft_consensus.cc:2258] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.019258 16947 ts_tablet_manager.cc:1875] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.019371 16949 ts_tablet_manager.cc:1875] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.019773 16948 ts_tablet_manager.cc:1875] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:56:46.021688 16949 ts_tablet_manager.cc:1888] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 19:56:46.021728 16947 ts_tablet_manager.cc:1888] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 19:56:46.021761 16949 log.cc:1196] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-0/wal/wals/9be119e1fbe74e5f84fa96d3eddadeef
I20240501 19:56:46.021759 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 16485
I20240501 19:56:46.021793 16947 log.cc:1196] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/9be119e1fbe74e5f84fa96d3eddadeef
I20240501 19:56:46.022008 16949 ts_tablet_manager.cc:1909] T 9be119e1fbe74e5f84fa96d3eddadeef P f30e604459744605bbdeb2fe62c9396b: Deleting consensus metadata
I20240501 19:56:46.022050 16947 ts_tablet_manager.cc:1909] T 9be119e1fbe74e5f84fa96d3eddadeef P e914b9526b3b43bda46fb11aeb9324b7: Deleting consensus metadata
I20240501 19:56:46.022797 16397 catalog_manager.cc:4830] TS e914b9526b3b43bda46fb11aeb9324b7 (127.8.168.66:34269): tablet 9be119e1fbe74e5f84fa96d3eddadeef (table test-table [id=d87d3345cdf24f7b9b3f72c854ecdf5e]) successfully deleted
I20240501 19:56:46.026268 16948 ts_tablet_manager.cc:1888] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 19:56:46.026351 16948 log.cc:1196] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/9be119e1fbe74e5f84fa96d3eddadeef
I20240501 19:56:46.026597 16948 ts_tablet_manager.cc:1909] T 9be119e1fbe74e5f84fa96d3eddadeef P 56d95af6390d40e7a395ff0a2d798726: Deleting consensus metadata
I20240501 19:56:46.027351 16397 catalog_manager.cc:4830] TS 56d95af6390d40e7a395ff0a2d798726 (127.8.168.67:34613): tablet 9be119e1fbe74e5f84fa96d3eddadeef (table test-table [id=d87d3345cdf24f7b9b3f72c854ecdf5e]) successfully deleted
W20240501 19:56:46.030953 16398 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.168.65:45667 (error 108)
I20240501 19:56:46.031140 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 16625
I20240501 19:56:46.037106 17021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:46.032994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:45667 (local address 127.0.0.1:43758)
0501 19:56:46.036895 (+ 3901us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:46.036973 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:45667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3813,"thread_start_us":56,"threads_started":1}
W20240501 19:56:46.037216 16398 catalog_manager.cc:4573] TS f30e604459744605bbdeb2fe62c9396b (127.8.168.65:45667): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9be119e1fbe74e5f84fa96d3eddadeef: Network error: Client connection negotiation failed: client connection to 127.8.168.65:45667: connect: Connection refused (error 111)
I20240501 19:56:46.037588 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 16765
I20240501 19:56:46.042963 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 16381
2024-05-01T19:56:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3191 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 19:56:46 dist-test-slave-tzct krb5kdc[17024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 19:56:46 dist-test-slave-tzct krb5kdc[17024](info): set up 2 sockets
May 01 19:56:46 dist-test-slave-tzct krb5kdc[17024](info): commencing operation
krb5kdc: starting...
W20240501 19:56:46.937477 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.873s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 19:56:47 dist-test-slave-tzct krb5kdc[17024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T19:56:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 19:56:47.045320 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:40055
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:40055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:47.147975 17037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:47.148213 17037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:47.148286 17037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:47.151330 17037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:47.151396 17037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:47.151427 17037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:47.151448 17037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:47.151479 17037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:47.155640 17037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:40055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:40055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:47.156503 17037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:47.157331 17037 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:47.164448 17045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:47.164944 17042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:47.165127 17043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:47.165614 17037 server_base.cc:1034] running on GCE node
I20240501 19:56:47.166060 17037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:47.166955 17037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:47.168112 17037 hybrid_clock.cc:648] HybridClock initialized: now 1714593407168100 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2024-05-01T19:56:47Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1128 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 19:56:47 dist-test-slave-tzct krb5kdc[17050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 19:56:47 dist-test-slave-tzct krb5kdc[17050](info): set up 2 sockets
May 01 19:56:47 dist-test-slave-tzct krb5kdc[17050](info): commencing operation
krb5kdc: starting...
W20240501 19:56:48.077617 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.884s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 19:56:48 dist-test-slave-tzct krb5kdc[17050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T19:56:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 19:56:48.189291 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33783
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:33783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:48.296417 17063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:48.296660 17063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:48.296747 17063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:48.299830 17063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:48.299894 17063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:48.299917 17063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:48.299957 17063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:48.299993 17063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:48.304491 17063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:33783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:48.305405 17063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:48.306237 17063 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:48.313205 17071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:48.313853 17068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:48.314257 17069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:48.314867 17063 server_base.cc:1034] running on GCE node
I20240501 19:56:48.315347 17063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:48.316268 17063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:48.317423 17063 hybrid_clock.cc:648] HybridClock initialized: now 1714593408317395 us; error 42 us; skew 500 ppm
May 01 19:56:48 dist-test-slave-tzct krb5kdc[17050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593408, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:48.320008 17063 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 19:56:48.320951 17063 webserver.cc:460] Webserver started at http://127.8.168.126:45791/ using document root <none> and password file <none>
I20240501 19:56:48.321468 17063 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:48.321524 17063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:48.321689 17063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:48.323408 17063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01"
format_stamp: "Formatted at 2024-05-01 19:56:48 on dist-test-slave-tzct"
server_key: "51fd52e187f3611b92a66e2cb80f1e9e"
server_key_iv: "5a236827dd689b5680cec4988f62ced5"
server_key_version: "encryptionkey@0"
I20240501 19:56:48.323861 17063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01"
format_stamp: "Formatted at 2024-05-01 19:56:48 on dist-test-slave-tzct"
server_key: "51fd52e187f3611b92a66e2cb80f1e9e"
server_key_iv: "5a236827dd689b5680cec4988f62ced5"
server_key_version: "encryptionkey@0"
I20240501 19:56:48.327244 17063 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:48.329481 17078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:48.330518 17063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:48.330639 17063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01"
format_stamp: "Formatted at 2024-05-01 19:56:48 on dist-test-slave-tzct"
server_key: "51fd52e187f3611b92a66e2cb80f1e9e"
server_key_iv: "5a236827dd689b5680cec4988f62ced5"
server_key_version: "encryptionkey@0"
I20240501 19:56:48.330736 17063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:48.347028 17063 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:48.349778 17063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:48.349951 17063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:48.356462 17063 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:33783
I20240501 19:56:48.356478 17130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:33783 every 8 connection(s)
I20240501 19:56:48.357445 17063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:56:48.360145 17131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:48.365007 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 17063
I20240501 19:56:48.365128 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:56:48.365118 17131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01: Bootstrap starting.
I20240501 19:56:48.365337 8865 external_mini_cluster.cc:1428] Setting key 7bd778cbadd94b31b88c4406922534b4
I20240501 19:56:48.367388 17131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:48.368064 17131 log.cc:824] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:48.369741 17131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01: No bootstrap required, opened a new log
May 01 19:56:48 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:48.372957 17131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33783 } }
I20240501 19:56:48.373188 17131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:48.373251 17131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbb8dcc1d504c6ab78813a16bb9ff01, State: Initialized, Role: FOLLOWER
I20240501 19:56:48.373692 17131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33783 } }
I20240501 19:56:48.373854 17131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:48.373921 17131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:48.374032 17131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:48.375020 17131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33783 } }
I20240501 19:56:48.375406 17131 leader_election.cc:302] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fbb8dcc1d504c6ab78813a16bb9ff01; no voters:
I20240501 19:56:48.375695 17131 leader_election.cc:288] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:48.375770 17136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:48.375953 17136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [term 1 LEADER]: Becoming Leader. State: Replica: 7fbb8dcc1d504c6ab78813a16bb9ff01, State: Running, Role: LEADER
I20240501 19:56:48.376279 17136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33783 } }
I20240501 19:56:48.376873 17131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:48.377451 17136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fbb8dcc1d504c6ab78813a16bb9ff01. Latest consensus state: current_term: 1 leader_uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33783 } } }
I20240501 19:56:48.378139 17136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:48.378289 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:48.366456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56400 (local address 127.8.168.126:33783)
0501 19:56:48.366905 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:48.366915 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:48.366949 (+ 34us) server_negotiation.cc:407] Connection header received
0501 19:56:48.367572 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:48.367596 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:48.367877 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:48.368204 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:48.369096 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:48.369881 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:48.370537 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:48.370796 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:48.373161 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:48.373185 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:48.373199 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:48.373238 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:48.375277 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:48.375685 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:48.375690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:48.375694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:48.375777 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:48.376041 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:48.376044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:48.376046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:48.376457 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:48.376677 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:48.376930 (+ 253us) server_negotiation.cc:299] Negotiation successful
0501 19:56:48.377440 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":157,"threads_started":1}
I20240501 19:56:48.378023 17137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33783 } } }
I20240501 19:56:48.378604 17137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [sys.catalog]: This master's current role is: LEADER
W20240501 19:56:48.381613 17152 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:56:48.381675 17152 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:56:48.381764 17145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:48.382751 17145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:48.387719 17145 catalog_manager.cc:1347] Generated new cluster ID: b17391730173431aa7b9ad6e169de504
I20240501 19:56:48.387784 17145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:48.408059 17145 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:48.409212 17145 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:48.415416 17145 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01: Generated new TSK 0
I20240501 19:56:48.416079 17145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
I20240501 19:56:48.443852 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33783
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:48.548740 17157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:48.548954 17157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:48.549011 17157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:48.552023 17157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:48.552085 17157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:48.552163 17157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:56:48.556257 17157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:48.557137 17157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:48.557946 17157 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:48.565785 17162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:48.566246 17163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:48.566779 17165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:48.567308 17157 server_base.cc:1034] running on GCE node
I20240501 19:56:48.567675 17157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:48.568197 17157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:48.569350 17157 hybrid_clock.cc:648] HybridClock initialized: now 1714593408569340 us; error 30 us; skew 500 ppm
May 01 19:56:48 dist-test-slave-tzct krb5kdc[17050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593408, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:48.571821 17157 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 19:56:48.572678 17157 webserver.cc:460] Webserver started at http://127.8.168.65:34347/ using document root <none> and password file <none>
I20240501 19:56:48.573194 17157 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:48.573247 17157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:48.573390 17157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:48.575071 17157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "cf3f21e18cc0482bb182e0d3d52ad69c"
format_stamp: "Formatted at 2024-05-01 19:56:48 on dist-test-slave-tzct"
server_key: "f515f1366584b91c0570090f8e69046e"
server_key_iv: "3fc2595678549a295babc4be5f012cca"
server_key_version: "encryptionkey@0"
I20240501 19:56:48.575501 17157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "cf3f21e18cc0482bb182e0d3d52ad69c"
format_stamp: "Formatted at 2024-05-01 19:56:48 on dist-test-slave-tzct"
server_key: "f515f1366584b91c0570090f8e69046e"
server_key_iv: "3fc2595678549a295babc4be5f012cca"
server_key_version: "encryptionkey@0"
I20240501 19:56:48.578876 17157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:48.581023 17172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:48.582022 17157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:56:48.582121 17157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "cf3f21e18cc0482bb182e0d3d52ad69c"
format_stamp: "Formatted at 2024-05-01 19:56:48 on dist-test-slave-tzct"
server_key: "f515f1366584b91c0570090f8e69046e"
server_key_iv: "3fc2595678549a295babc4be5f012cca"
server_key_version: "encryptionkey@0"
I20240501 19:56:48.582206 17157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:48.610029 17157 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:48.612860 17157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:48.613018 17157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:48.613554 17157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:48.614342 17157 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:48.614389 17157 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:48.614442 17157 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:48.614470 17157 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:48.623015 17157 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:38143
I20240501 19:56:48.623064 17285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:38143 every 8 connection(s)
I20240501 19:56:48.623912 17157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 19:56:48.629284 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 17157
I20240501 19:56:48.629392 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 19:56:48.629621 8865 external_mini_cluster.cc:1428] Setting key df3fdb1c4fae93362f5a2325a4432e44
May 01 19:56:48 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593408, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:48.636371 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:48.625659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:52165 (local address 127.8.168.126:33783)
0501 19:56:48.625822 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:48.625826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:48.626573 (+ 747us) server_negotiation.cc:407] Connection header received
0501 19:56:48.627421 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:48.627425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:48.627482 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:48.627563 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:48.629195 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:48.629775 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:48.630424 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:48.630600 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:48.632951 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:48.632969 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:48.632972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:48.632996 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:48.634433 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:48.634940 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:48.634943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:48.634945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:48.634998 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:48.635313 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:48.635316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:48.635318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:48.635475 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:48.635561 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:48.636066 (+ 505us) server_negotiation.cc:299] Negotiation successful
0501 19:56:48.636207 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20240501 19:56:48.637017 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:48.625905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:52165)
0501 19:56:48.626339 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:48.626375 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:48.627216 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:48.627731 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:48.627738 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:48.628106 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:48.629050 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:48.629065 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:48.629890 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:48.629893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:48.630305 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:48.630313 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:48.630529 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:48.631139 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:48.631165 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:48.632799 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:48.634549 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:48.634554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:48.634567 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:48.634813 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:48.635116 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:48.635120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:48.635122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:48.635226 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:48.635581 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:48.635587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:48.635829 (+ 242us) client_negotiation.cc:769] Sending connection context
0501 19:56:48.636012 (+ 183us) client_negotiation.cc:240] Negotiation successful
0501 19:56:48.636292 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.8.168.66@KRBTEST.COM; defaulting to no policy
I20240501 19:56:48.638254 17286 heartbeater.cc:347] Connected to a master server at 127.8.168.126:33783
I20240501 19:56:48.638537 17286 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:48.639119 17286 heartbeater.cc:511] Master 127.8.168.126:33783 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.66@KRBTEST.COM" created.
I20240501 19:56:48.640655 17095 ts_manager.cc:194] Registered new tserver with Master: cf3f21e18cc0482bb182e0d3d52ad69c (127.8.168.65:38143)
I20240501 19:56:48.641806 17095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:52165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
I20240501 19:56:48.655854 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33783
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:48.758486 17291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:48.758705 17291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:48.758760 17291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:48.761732 17291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:48.761793 17291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:48.761868 17291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:56:48.765991 17291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:48.766806 17291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:48.767592 17291 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:48.775625 17296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:48.776109 17297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:48.776285 17299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:48.776755 17291 server_base.cc:1034] running on GCE node
I20240501 19:56:48.777137 17291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:48.777698 17291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:48.778856 17291 hybrid_clock.cc:648] HybridClock initialized: now 1714593408778838 us; error 33 us; skew 500 ppm
May 01 19:56:48 dist-test-slave-tzct krb5kdc[17050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593408, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:48.781405 17291 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 19:56:48.782208 17291 webserver.cc:460] Webserver started at http://127.8.168.66:40189/ using document root <none> and password file <none>
I20240501 19:56:48.782714 17291 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:48.782771 17291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:48.782931 17291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:48.784627 17291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "bdb55124c124406e95b82672fdf174e3"
format_stamp: "Formatted at 2024-05-01 19:56:48 on dist-test-slave-tzct"
server_key: "abd45b2b41c9d4152154b6dad5977a5c"
server_key_iv: "d2b1002993acac63674c86d4c3ee778c"
server_key_version: "encryptionkey@0"
I20240501 19:56:48.785149 17291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "bdb55124c124406e95b82672fdf174e3"
format_stamp: "Formatted at 2024-05-01 19:56:48 on dist-test-slave-tzct"
server_key: "abd45b2b41c9d4152154b6dad5977a5c"
server_key_iv: "d2b1002993acac63674c86d4c3ee778c"
server_key_version: "encryptionkey@0"
I20240501 19:56:48.788231 17291 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 19:56:48.790320 17306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:48.791276 17291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:48.791383 17291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "bdb55124c124406e95b82672fdf174e3"
format_stamp: "Formatted at 2024-05-01 19:56:48 on dist-test-slave-tzct"
server_key: "abd45b2b41c9d4152154b6dad5977a5c"
server_key_iv: "d2b1002993acac63674c86d4c3ee778c"
server_key_version: "encryptionkey@0"
I20240501 19:56:48.791460 17291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:48.820518 17291 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:48.823295 17291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:48.823475 17291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:48.824023 17291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:48.824873 17291 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:48.824921 17291 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:48.824977 17291 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:48.825086 17291 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:48.834401 17419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:45647 every 8 connection(s)
I20240501 19:56:48.834404 17291 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:45647
I20240501 19:56:48.835480 17291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 19:56:48.841212 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 17291
I20240501 19:56:48.841313 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 19:56:48.841579 8865 external_mini_cluster.cc:1428] Setting key 81fe71016be3fe3f0b7e9cf0ffbd5076
May 01 19:56:48 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593408, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:48.847977 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:48.837240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:53737 (local address 127.8.168.126:33783)
0501 19:56:48.837385 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:48.837389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:48.838080 (+ 691us) server_negotiation.cc:407] Connection header received
0501 19:56:48.838887 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:48.838891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:48.838948 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:48.839022 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:48.840627 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:48.841237 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:48.842063 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:48.842263 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:48.844778 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:48.844796 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:48.844799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:48.844823 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:48.846183 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:48.846653 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:48.846656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:48.846658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:48.846703 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:48.846958 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:48.846961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:48.846963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:48.847117 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:48.847200 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:48.847695 (+ 495us) server_negotiation.cc:299] Negotiation successful
0501 19:56:48.847817 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20240501 19:56:48.848599 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:48.837495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:53737)
0501 19:56:48.837932 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:48.837967 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:48.838664 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:48.839178 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:48.839186 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:48.839571 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:48.840480 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:48.840493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:48.841373 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:48.841377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:48.841933 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:48.841945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:48.842178 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:48.842869 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:48.842905 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:48.844602 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:48.846298 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:48.846304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:48.846315 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:48.846549 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:48.846785 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:48.846788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:48.846790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:48.846874 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:48.847219 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:48.847226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:48.847458 (+ 232us) client_negotiation.cc:769] Sending connection context
0501 19:56:48.847642 (+ 184us) client_negotiation.cc:240] Negotiation successful
0501 19:56:48.847912 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.8.168.67@KRBTEST.COM; defaulting to no policy
I20240501 19:56:48.849670 17420 heartbeater.cc:347] Connected to a master server at 127.8.168.126:33783
I20240501 19:56:48.849879 17420 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:48.850374 17420 heartbeater.cc:511] Master 127.8.168.126:33783 requested a full tablet report, sending...
I20240501 19:56:48.851416 17095 ts_manager.cc:194] Registered new tserver with Master: bdb55124c124406e95b82672fdf174e3 (127.8.168.66:45647)
I20240501 19:56:48.851992 17095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:53737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
I20240501 19:56:48.867400 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33783
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:48.970794 17425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:48.971009 17425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:48.971081 17425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:48.974190 17425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:48.974247 17425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:48.974340 17425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:56:48.978377 17425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:48.979190 17425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:48.979972 17425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:48.987891 17430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:48.987958 17433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:48.988549 17431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:48.989027 17425 server_base.cc:1034] running on GCE node
I20240501 19:56:48.989326 17425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:48.989781 17425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:48.990927 17425 hybrid_clock.cc:648] HybridClock initialized: now 1714593408990918 us; error 26 us; skew 500 ppm
May 01 19:56:48 dist-test-slave-tzct krb5kdc[17050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593408, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:48.993485 17425 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 19:56:48.994341 17425 webserver.cc:460] Webserver started at http://127.8.168.67:37009/ using document root <none> and password file <none>
I20240501 19:56:48.994830 17425 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:48.994887 17425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:48.995052 17425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:48.996795 17425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "aad82ded81914782b35ce212ed2143da"
format_stamp: "Formatted at 2024-05-01 19:56:48 on dist-test-slave-tzct"
server_key: "58e18af32a53edb08473aebce70fe789"
server_key_iv: "437e577aabc7d5fe34f4f383918006c7"
server_key_version: "encryptionkey@0"
I20240501 19:56:48.997224 17425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "aad82ded81914782b35ce212ed2143da"
format_stamp: "Formatted at 2024-05-01 19:56:48 on dist-test-slave-tzct"
server_key: "58e18af32a53edb08473aebce70fe789"
server_key_iv: "437e577aabc7d5fe34f4f383918006c7"
server_key_version: "encryptionkey@0"
I20240501 19:56:49.000239 17425 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240501 19:56:49.002326 17440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:49.003247 17425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:49.003337 17425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "aad82ded81914782b35ce212ed2143da"
format_stamp: "Formatted at 2024-05-01 19:56:48 on dist-test-slave-tzct"
server_key: "58e18af32a53edb08473aebce70fe789"
server_key_iv: "437e577aabc7d5fe34f4f383918006c7"
server_key_version: "encryptionkey@0"
I20240501 19:56:49.003407 17425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:49.030900 17425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:49.033764 17425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:49.033895 17425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:49.034372 17425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:49.035131 17425 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:49.035177 17425 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:49.035214 17425 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:49.035233 17425 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:49.043088 17425 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:33251
I20240501 19:56:49.043133 17553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:33251 every 8 connection(s)
I20240501 19:56:49.043954 17425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593408, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:56:49.052770 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 17425
I20240501 19:56:49.052870 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 19:56:49.053128 8865 external_mini_cluster.cc:1428] Setting key 72cba0d90079c79aae598496cd25cda3
I20240501 19:56:49.055377 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:49.045573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:57847 (local address 127.8.168.126:33783)
0501 19:56:49.045710 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:49.045713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:49.046325 (+ 612us) server_negotiation.cc:407] Connection header received
0501 19:56:49.047119 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:49.047123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:49.047180 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:49.047251 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:49.048798 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.049298 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.049919 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.050106 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.052062 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:49.052080 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:49.052089 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:49.052112 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:49.053486 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:49.053994 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:49.053998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:49.054000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:49.054052 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:49.054337 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:49.054340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:49.054342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:49.054517 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:49.054592 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:49.055161 (+ 569us) server_negotiation.cc:299] Negotiation successful
0501 19:56:49.055264 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20240501 19:56:49.055933 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:49.045813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:57847)
0501 19:56:49.046189 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:49.046222 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:49.046902 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:49.047414 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:49.047422 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:49.047779 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:49.048647 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.048660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.049413 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.049416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:49.049807 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.049815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.050010 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.050568 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:49.050601 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:49.051917 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:56:49.053598 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:49.053603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:49.053611 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:49.053896 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:49.054144 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:56:49.054146 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:56:49.054148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:56:49.054244 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:56:49.054613 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:56:49.054619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:56:49.054896 (+ 277us) client_negotiation.cc:769] Sending connection context
0501 19:56:49.055116 (+ 220us) client_negotiation.cc:240] Negotiation successful
0501 19:56:49.055343 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":94,"threads_started":1}
I20240501 19:56:49.056919 17554 heartbeater.cc:347] Connected to a master server at 127.8.168.126:33783
I20240501 19:56:49.057129 17554 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:49.057611 17554 heartbeater.cc:511] Master 127.8.168.126:33783 requested a full tablet report, sending...
I20240501 19:56:49.058493 17095 ts_manager.cc:194] Registered new tserver with Master: aad82ded81914782b35ce212ed2143da (127.8.168.67:33251)
I20240501 19:56:49.058990 17095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:57847
I20240501 19:56:49.066661 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.65.keytab.
Entry for principal oryx/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.65.keytab.
WARNING: no policy specified for oryx/127.8.168.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.168.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.66.keytab.
Entry for principal oryx/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.66.keytab.
WARNING: no policy specified for oryx/127.8.168.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.168.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.67.keytab.
Entry for principal oryx/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.67.keytab.
I20240501 19:56:49.129539 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 17157
I20240501 19:56:49.134246 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 17291
I20240501 19:56:49.138367 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 17425
I20240501 19:56:49.142571 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 17063
I20240501 19:56:49.147497 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33783
--webserver_interface=127.8.168.126
--webserver_port=45791
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:33783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:56:49.249454 17563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:49.249666 17563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:49.249734 17563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:49.252681 17563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:49.252760 17563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:49.252792 17563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:49.252816 17563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:49.252853 17563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:49.256866 17563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:33783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:49.257663 17563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:49.258450 17563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:49.265213 17569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:49.265748 17568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:49.266288 17571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:49.266712 17563 server_base.cc:1034] running on GCE node
I20240501 19:56:49.267201 17563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:49.268069 17563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:49.269218 17563 hybrid_clock.cc:648] HybridClock initialized: now 1714593409269190 us; error 42 us; skew 500 ppm
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593409, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:49.271718 17563 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 19:56:49.272626 17563 webserver.cc:460] Webserver started at http://127.8.168.126:45791/ using document root <none> and password file <none>
I20240501 19:56:49.273178 17563 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:49.273250 17563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:49.276549 17563 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:49.278458 17578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:49.279402 17563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:49.279513 17563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01"
format_stamp: "Formatted at 2024-05-01 19:56:48 on dist-test-slave-tzct"
server_key: "51fd52e187f3611b92a66e2cb80f1e9e"
server_key_iv: "5a236827dd689b5680cec4988f62ced5"
server_key_version: "encryptionkey@0"
I20240501 19:56:49.279901 17563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:49.299808 17563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:49.302867 17563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:49.303035 17563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:49.309533 17563 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:33783
I20240501 19:56:49.309563 17630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:33783 every 8 connection(s)
I20240501 19:56:49.310688 17563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:56:49.312279 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 17563
I20240501 19:56:49.312805 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:38143
--local_ip_for_outbound_sockets=127.8.168.65
--tserver_master_addrs=127.8.168.126:33783
--webserver_port=34347
--webserver_interface=127.8.168.65
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.168.65
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 19:56:49.315445 17631 sys_catalog.cc:263] Verifying existing consensus state
I20240501 19:56:49.316936 17631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01: Bootstrap starting.
I20240501 19:56:49.332317 17631 log.cc:824] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:49.338480 17631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:49.338809 17631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01: Bootstrap complete.
I20240501 19:56:49.342092 17631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33783 } }
I20240501 19:56:49.342408 17631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbb8dcc1d504c6ab78813a16bb9ff01, State: Initialized, Role: FOLLOWER
I20240501 19:56:49.342875 17631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33783 } }
I20240501 19:56:49.343003 17631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:49.343060 17631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:49.343161 17631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:49.344687 17631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33783 } }
I20240501 19:56:49.345041 17631 leader_election.cc:302] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fbb8dcc1d504c6ab78813a16bb9ff01; no voters:
I20240501 19:56:49.345309 17631 leader_election.cc:288] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240501 19:56:49.345479 17635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:49.345782 17635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [term 2 LEADER]: Becoming Leader. State: Replica: 7fbb8dcc1d504c6ab78813a16bb9ff01, State: Running, Role: LEADER
I20240501 19:56:49.346120 17635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33783 } }
I20240501 19:56:49.346378 17631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:49.347350 17636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33783 } } }
I20240501 19:56:49.347489 17636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:49.348084 17637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fbb8dcc1d504c6ab78813a16bb9ff01. Latest consensus state: current_term: 2 leader_uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbb8dcc1d504c6ab78813a16bb9ff01" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33783 } } }
I20240501 19:56:49.348158 17637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:49.350865 17652 catalog_manager.cc:1259] Loaded cluster ID: b17391730173431aa7b9ad6e169de504
I20240501 19:56:49.350924 17652 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01: loading cluster ID for follower catalog manager: success
I20240501 19:56:49.353355 17652 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01: acquiring CA information for follower catalog manager: success
I20240501 19:56:49.354714 17652 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:56:49.354916 17644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:49.355868 17644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:49.356278 17644 catalog_manager.cc:1259] Loaded cluster ID: b17391730173431aa7b9ad6e169de504
I20240501 19:56:49.356333 17644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:49.356938 17644 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:49.357385 17644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fbb8dcc1d504c6ab78813a16bb9ff01: Loaded TSK: 0
I20240501 19:56:49.358067 17644 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240501 19:56:49.418515 17633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:49.418720 17633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:49.418776 17633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:49.421792 17633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:49.421849 17633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:49.421936 17633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:56:49.425926 17633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.65.keytab
--principal=oryx/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:38143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=34347
--tserver_master_addrs=127.8.168.126:33783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:49.426751 17633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:49.427577 17633 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:49.435599 17658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:49.436057 17659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:49.436640 17661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:49.437158 17633 server_base.cc:1034] running on GCE node
I20240501 19:56:49.437525 17633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:49.438102 17633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:49.439268 17633 hybrid_clock.cc:648] HybridClock initialized: now 1714593409439242 us; error 45 us; skew 500 ppm
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593409, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:49.441979 17633 init.cc:377] Logged in from keytab as oryx/127.8.168.65@KRBTEST.COM (short username oryx)
I20240501 19:56:49.442932 17633 webserver.cc:460] Webserver started at http://127.8.168.65:34347/ using document root <none> and password file <none>
I20240501 19:56:49.443447 17633 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:49.443519 17633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:49.446898 17633 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:49.448827 17668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:49.449860 17633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:49.449962 17633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "cf3f21e18cc0482bb182e0d3d52ad69c"
format_stamp: "Formatted at 2024-05-01 19:56:48 on dist-test-slave-tzct"
server_key: "f515f1366584b91c0570090f8e69046e"
server_key_iv: "3fc2595678549a295babc4be5f012cca"
server_key_version: "encryptionkey@0"
I20240501 19:56:49.450363 17633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:49.487375 17633 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:49.490339 17633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:49.490504 17633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:49.491045 17633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:49.491819 17633 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:49.491868 17633 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:49.491921 17633 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:49.491969 17633 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:49.501797 17633 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:38143
I20240501 19:56:49.501839 17781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:38143 every 8 connection(s)
I20240501 19:56:49.502789 17633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 19:56:49.509286 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 17633
I20240501 19:56:49.509678 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:45647
--local_ip_for_outbound_sockets=127.8.168.66
--tserver_master_addrs=127.8.168.126:33783
--webserver_port=40189
--webserver_interface=127.8.168.66
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.168.66
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
W20240501 19:56:49.513617 17784 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:49.504607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:35799 (local address 127.8.168.126:33783)
0501 19:56:49.505093 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:49.505105 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:49.505274 (+ 169us) server_negotiation.cc:407] Connection header received
0501 19:56:49.506316 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:49.506325 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:49.506577 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:49.506885 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:49.508462 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.509274 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.509963 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.510242 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.513083 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:35799: BlockingRecv error: recv got EOF from 127.8.168.65:35799 (error 108)
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":116,"threads_started":1}
I20240501 19:56:49.513620 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:49.504642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:35799)
0501 19:56:49.505125 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:49.505167 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:49.505916 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:49.506970 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:49.506979 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:49.507414 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:49.508307 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.508320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.509400 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.509404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:49.509825 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.509834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.510018 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.510583 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:49.510608 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:49.512897 (+ 2289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":107,"threads_started":1}
W20240501 19:56:49.514022 17782 heartbeater.cc:644] Failed to heartbeat to 127.8.168.126:33783 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.168.126:33783: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:49.518620 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:49.514417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:33097)
0501 19:56:49.514534 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:49.514547 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:49.514617 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:49.514853 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:49.514856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:49.514993 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:49.515170 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.515176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.515844 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.515846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:49.516215 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.516223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.516319 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.516938 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:49.516950 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:49.518498 (+ 1548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20240501 19:56:49.518841 17784 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:49.514545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:33097 (local address 127.8.168.126:33783)
0501 19:56:49.514677 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:49.514681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:49.514693 (+ 12us) server_negotiation.cc:407] Connection header received
0501 19:56:49.514721 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:49.514723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:49.514757 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:49.514819 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:49.515281 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.515741 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.516321 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.516500 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.518646 (+ 2146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:33097: BlockingRecv error: recv got EOF from 127.8.168.65:33097 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:49.523268 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:49.519166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:43923)
0501 19:56:49.519284 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:49.519295 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:49.519354 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:49.519588 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:49.519590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:49.519696 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:49.519836 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.519840 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.520785 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.520787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:49.521126 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.521132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.521211 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.521637 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:49.521649 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:49.523142 (+ 1493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20240501 19:56:49.523334 17784 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:49.519177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:43923 (local address 127.8.168.126:33783)
0501 19:56:49.519318 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:49.519322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:49.519338 (+ 16us) server_negotiation.cc:407] Connection header received
0501 19:56:49.519421 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:49.519424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:49.519491 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:49.519582 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:49.519939 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.520668 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.521224 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.521406 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.523246 (+ 1840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:43923: BlockingRecv error: recv got EOF from 127.8.168.65:43923 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20240501 19:56:49.523469 17782 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240501 19:56:49.616221 17786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:49.616425 17786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:49.616478 17786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:49.619484 17786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:49.619546 17786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:49.619634 17786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:56:49.623661 17786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.66.keytab
--principal=oryx/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:45647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=40189
--tserver_master_addrs=127.8.168.126:33783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:49.624531 17786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:49.625442 17786 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:49.633339 17791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:49.633685 17794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:49.633857 17792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:49.634429 17786 server_base.cc:1034] running on GCE node
I20240501 19:56:49.634766 17786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:49.635252 17786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:49.636400 17786 hybrid_clock.cc:648] HybridClock initialized: now 1714593409636379 us; error 32 us; skew 500 ppm
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593409, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:49.638929 17786 init.cc:377] Logged in from keytab as oryx/127.8.168.66@KRBTEST.COM (short username oryx)
I20240501 19:56:49.639787 17786 webserver.cc:460] Webserver started at http://127.8.168.66:40189/ using document root <none> and password file <none>
I20240501 19:56:49.640280 17786 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:49.640342 17786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:49.643677 17786 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:49.645520 17801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:49.646557 17786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:49.646661 17786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "bdb55124c124406e95b82672fdf174e3"
format_stamp: "Formatted at 2024-05-01 19:56:48 on dist-test-slave-tzct"
server_key: "abd45b2b41c9d4152154b6dad5977a5c"
server_key_iv: "d2b1002993acac63674c86d4c3ee778c"
server_key_version: "encryptionkey@0"
I20240501 19:56:49.647102 17786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:49.661235 17786 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:49.663743 17786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:49.663872 17786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:49.664350 17786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:49.665151 17786 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:49.665199 17786 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:49.665238 17786 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:49.665254 17786 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:49.673388 17786 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:45647
I20240501 19:56:49.673416 17914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:45647 every 8 connection(s)
I20240501 19:56:49.674397 17786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 19:56:49.675233 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 17786
I20240501 19:56:49.675639 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:33251
--local_ip_for_outbound_sockets=127.8.168.67
--tserver_master_addrs=127.8.168.126:33783
--webserver_port=37009
--webserver_interface=127.8.168.67
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.168.67
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
W20240501 19:56:49.685040 17784 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:49.676030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:60489 (local address 127.8.168.126:33783)
0501 19:56:49.676190 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:49.676195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:49.676932 (+ 737us) server_negotiation.cc:407] Connection header received
0501 19:56:49.677822 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:49.677825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:49.677877 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:49.677947 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:49.680056 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.680542 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.681472 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.681610 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.684940 (+ 3330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:60489: BlockingRecv error: recv got EOF from 127.8.168.66:60489 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20240501 19:56:49.685595 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:49.676299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:60489)
0501 19:56:49.676770 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:49.676817 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:49.677592 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:49.678140 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:49.678150 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:49.678655 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:49.679898 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.679918 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.680666 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.680670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:49.681319 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.681331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.681593 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.682191 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:49.682230 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:49.684836 (+ 2606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":96,"threads_started":1}
W20240501 19:56:49.685998 17915 heartbeater.cc:644] Failed to heartbeat to 127.8.168.126:33783 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.168.126:33783: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:49.690491 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:49.686374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:55751)
0501 19:56:49.686484 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:49.686498 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:49.686559 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:49.686807 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:49.686810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:49.686943 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:49.687101 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.687107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.687756 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.687759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:49.688101 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.688108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.688201 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.688798 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:49.688812 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:49.690357 (+ 1545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20240501 19:56:49.690546 17784 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:49.686505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:55751 (local address 127.8.168.126:33783)
0501 19:56:49.686637 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:49.686641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:49.686652 (+ 11us) server_negotiation.cc:407] Connection header received
0501 19:56:49.686679 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:49.686681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:49.686714 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:49.686777 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:49.687213 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.687654 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.688205 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.688374 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.690459 (+ 2085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:55751: BlockingRecv error: recv got EOF from 127.8.168.66:55751 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:49.694811 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:49.691001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:45299)
0501 19:56:49.691100 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:49.691113 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:49.691181 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:49.691389 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:49.691392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:49.691511 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:49.691655 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.691661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.692293 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.692296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:49.692654 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.692677 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.692801 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.693303 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:49.693314 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:49.694705 (+ 1391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20240501 19:56:49.694917 17784 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:49.691068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:45299 (local address 127.8.168.126:33783)
0501 19:56:49.691173 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:49.691176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:49.691187 (+ 11us) server_negotiation.cc:407] Connection header received
0501 19:56:49.691261 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:49.691264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:49.691299 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:49.691360 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:49.691761 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.692196 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.692788 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.692952 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.694820 (+ 1868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:45299: BlockingRecv error: recv got EOF from 127.8.168.66:45299 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20240501 19:56:49.695145 17915 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240501 19:56:49.780180 17918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:49.780382 17918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:49.780450 17918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:49.783512 17918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:56:49.783573 17918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:49.783679 17918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:56:49.787703 17918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/krb5kdc/oryx_127.8.168.67.keytab
--principal=oryx/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:33251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=37009
--tserver_master_addrs=127.8.168.126:33783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:49.788538 17918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:49.789379 17918 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:49.797230 17926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:49.797384 17924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:49.797900 17923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:49.798465 17918 server_base.cc:1034] running on GCE node
I20240501 19:56:49.798820 17918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:49.799350 17918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:49.800489 17918 hybrid_clock.cc:648] HybridClock initialized: now 1714593409800472 us; error 35 us; skew 500 ppm
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593409, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:56:49.803123 17918 init.cc:377] Logged in from keytab as oryx/127.8.168.67@KRBTEST.COM (short username oryx)
I20240501 19:56:49.803977 17918 webserver.cc:460] Webserver started at http://127.8.168.67:37009/ using document root <none> and password file <none>
I20240501 19:56:49.804487 17918 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:49.804556 17918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:49.807996 17918 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:49.809844 17933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:49.810783 17918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:49.810882 17918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "aad82ded81914782b35ce212ed2143da"
format_stamp: "Formatted at 2024-05-01 19:56:48 on dist-test-slave-tzct"
server_key: "58e18af32a53edb08473aebce70fe789"
server_key_iv: "437e577aabc7d5fe34f4f383918006c7"
server_key_version: "encryptionkey@0"
I20240501 19:56:49.811275 17918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:49.833575 17918 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:49.836024 17918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:49.836184 17918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:49.836683 17918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:49.837435 17918 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:49.837482 17918 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:49.837533 17918 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:49.837563 17918 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:49.845623 17918 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:33251
I20240501 19:56:49.845703 18046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:33251 every 8 connection(s)
I20240501 19:56:49.846611 17918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 19:56:49.851107 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 17918
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
W20240501 19:56:49.856303 17784 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:49.848220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:43829 (local address 127.8.168.126:33783)
0501 19:56:49.848344 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:49.848348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:49.849138 (+ 790us) server_negotiation.cc:407] Connection header received
0501 19:56:49.849888 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:49.849891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:49.849970 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:49.850055 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:49.851703 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.852329 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.853029 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.853202 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.856179 (+ 2977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.67:43829: BlockingRecv error: recv got EOF from 127.8.168.67:43829 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20240501 19:56:49.856673 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:49.848585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:43829)
0501 19:56:49.849004 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:49.849037 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:49.849691 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:49.850224 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:49.850232 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:49.850568 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:49.851513 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.851532 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.852458 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.852461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:49.852915 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.852924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.853124 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.853693 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:49.853718 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:49.856039 (+ 2321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":123,"threads_started":1}
W20240501 19:56:49.857059 18047 heartbeater.cc:644] Failed to heartbeat to 127.8.168.126:33783 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.168.126:33783: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
I20240501 19:56:49.859210 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:49.851918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56402 (local address 127.8.168.126:33783)
0501 19:56:49.852155 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:49.852159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:49.852170 (+ 11us) server_negotiation.cc:407] Connection header received
0501 19:56:49.852200 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:49.852202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:49.852237 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:49.852313 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:49.853151 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.853688 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.854355 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.854531 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.855489 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:56:49.855509 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:56:49.855521 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:56:49.855558 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:56:49.857410 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:49.857875 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:49.857881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:49.857885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:49.857960 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:56:49.858179 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:56:49.858182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:56:49.858184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:56:49.858450 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:56:49.858657 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0501 19:56:49.858901 (+ 244us) server_negotiation.cc:299] Negotiation successful
0501 19:56:49.859096 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":71,"threads_started":1}
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:49.861713 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:49.857431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:51751)
0501 19:56:49.857566 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:49.857578 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:49.857645 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:49.857855 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:49.857857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:49.857989 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:49.858151 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.858157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.858828 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.858832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:49.859230 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.859238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.859330 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.859854 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:49.859865 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:49.861633 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20240501 19:56:49.861820 17784 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:49.857531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:51751 (local address 127.8.168.126:33783)
0501 19:56:49.857644 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:49.857648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:49.857660 (+ 12us) server_negotiation.cc:407] Connection header received
0501 19:56:49.857724 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:49.857727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:49.857764 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:49.857824 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:49.858258 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.858729 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.859334 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.859513 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.861725 (+ 2212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.67:51751: BlockingRecv error: recv got EOF from 127.8.168.67:51751 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:49 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:49.865787 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:49.862135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:35619)
0501 19:56:49.862235 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:49.862248 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:49.862301 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:49.862465 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:49.862467 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:49.862570 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:49.862714 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.862719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.863421 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.863424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:49.863825 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:49.863833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.863918 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.864313 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:49.864325 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:49.865709 (+ 1384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20240501 19:56:49.865914 17784 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:49.862201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:35619 (local address 127.8.168.126:33783)
0501 19:56:49.862305 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:49.862307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:49.862318 (+ 11us) server_negotiation.cc:407] Connection header received
0501 19:56:49.862358 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:49.862360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:49.862390 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:49.862443 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:49.862841 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.863319 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:49.863947 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:49.864084 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:49.865815 (+ 1731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.67:35619: BlockingRecv error: recv got EOF from 127.8.168.67:35619 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20240501 19:56:49.865958 18047 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
May 01 19:56:50 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:50 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:50.532485 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:50.524065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:34781)
0501 19:56:50.524322 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:50.524335 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:50.524421 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:50.524644 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:50.524647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:50.524858 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:50.525086 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:50.525093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:50.525855 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:50.525857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:50.526229 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:50.526235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:50.526321 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:50.530628 (+ 4307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:50.530642 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:50.532360 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
W20240501 19:56:50.532603 18053 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:50.524205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:34781 (local address 127.8.168.126:33783)
0501 19:56:50.524417 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:50.524420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:50.524433 (+ 13us) server_negotiation.cc:407] Connection header received
0501 19:56:50.524499 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:50.524501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:50.524544 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:50.524614 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:50.525232 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:50.525764 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:50.526339 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:50.526493 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:50.532488 (+ 5995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:34781: BlockingRecv error: recv got EOF from 127.8.168.65:34781 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":60,"threads_started":1}
May 01 19:56:50 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:50 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:50.700309 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:50.695726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:40759)
0501 19:56:50.695982 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:50.695995 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:50.696089 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:50.696305 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:50.696308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:50.696473 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:50.696705 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:50.696712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:50.697440 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:50.697442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:50.697827 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:50.697833 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:50.697915 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:50.698438 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:50.698451 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:50.700156 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
W20240501 19:56:50.700418 18053 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:50.695798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:40759 (local address 127.8.168.126:33783)
0501 19:56:50.695941 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:50.695946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:50.696058 (+ 112us) server_negotiation.cc:407] Connection header received
0501 19:56:50.696163 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:50.696165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:50.696211 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:50.696276 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:50.696853 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:50.697355 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:50.697914 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:50.698046 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:50.700306 (+ 2260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:40759: BlockingRecv error: recv got EOF from 127.8.168.66:40759 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
May 01 19:56:50 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:50 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:50.871126 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:50.866457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:48875)
0501 19:56:50.866737 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:50.866749 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:50.866850 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:50.867117 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:50.867120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:50.867300 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:50.867530 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:50.867536 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:50.868240 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:50.868243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:50.868629 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:50.868635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:50.868755 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:50.869277 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:50.869293 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:50.870967 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
W20240501 19:56:50.871246 18053 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:50.866535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:48875 (local address 127.8.168.126:33783)
0501 19:56:50.866682 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:50.866686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:50.866816 (+ 130us) server_negotiation.cc:407] Connection header received
0501 19:56:50.866936 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:50.866940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:50.866983 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:50.867075 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:50.867648 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:50.868136 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:50.868758 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:50.868937 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:50.871131 (+ 2194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.67:48875: BlockingRecv error: recv got EOF from 127.8.168.67:48875 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
May 01 19:56:51 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:51 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:51.537847 18056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:51.533292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:49887)
0501 19:56:51.533548 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:51.533561 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:51.533647 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:51.533874 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:51.533876 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:51.534043 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:51.534249 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:51.534255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:51.534931 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:51.534933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:51.535308 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:51.535313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:51.535396 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:51.535910 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:51.535922 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:51.537700 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":71,"threads_started":1}
W20240501 19:56:51.537971 18057 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:51.533370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:49887 (local address 127.8.168.126:33783)
0501 19:56:51.533588 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:51.533591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:51.533604 (+ 13us) server_negotiation.cc:407] Connection header received
0501 19:56:51.533724 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:51.533726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:51.533770 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:51.533848 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:51.534346 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:51.534850 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:51.535382 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:51.535502 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:51.537859 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:49887: BlockingRecv error: recv got EOF from 127.8.168.65:49887 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":64,"threads_started":1}
May 01 19:56:51 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:51 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:51.705407 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:51.700966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:33563)
0501 19:56:51.701253 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:51.701266 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:51.701358 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:51.701583 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:51.701585 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:51.701755 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:51.701944 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:51.701950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:51.702607 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:51.702610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:51.702968 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:51.702974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:51.703056 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:51.703549 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:51.703561 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:51.705261 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":98,"threads_started":1}
W20240501 19:56:51.705523 18057 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:51.701036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:33563 (local address 127.8.168.126:33783)
0501 19:56:51.701176 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:51.701180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:51.701330 (+ 150us) server_negotiation.cc:407] Connection header received
0501 19:56:51.701430 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:51.701433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:51.701480 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:51.701560 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:51.702034 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:51.702530 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:51.703044 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:51.703165 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:51.705408 (+ 2243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:33563: BlockingRecv error: recv got EOF from 127.8.168.66:33563 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
May 01 19:56:51 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:51 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:51.876253 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:51.871738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:37775)
0501 19:56:51.872022 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:51.872036 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:51.872141 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:51.872377 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:51.872380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:51.872560 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:51.872784 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:51.872791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:51.873480 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:51.873483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:51.873850 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:51.873857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:51.873954 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:51.874478 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:51.874491 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:51.876117 (+ 1626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
W20240501 19:56:51.876346 18057 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:51.871881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:37775 (local address 127.8.168.126:33783)
0501 19:56:51.872005 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:51.872010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:51.872102 (+ 92us) server_negotiation.cc:407] Connection header received
0501 19:56:51.872230 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:51.872234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:51.872277 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:51.872340 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:51.872901 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:51.873377 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:51.873960 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:51.874135 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:51.876252 (+ 2117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.67:37775: BlockingRecv error: recv got EOF from 127.8.168.67:37775 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
May 01 19:56:52 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:52 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:52.543151 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:52.538604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:39741)
0501 19:56:52.538870 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:52.538882 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:52.538969 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:52.539206 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:52.539208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:52.539377 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:52.539569 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:52.539574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:52.540325 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:52.540327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:52.540699 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:52.540705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:52.540828 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:52.541349 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:52.541360 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:52.543023 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
W20240501 19:56:52.543292 18061 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:52.538706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:39741 (local address 127.8.168.126:33783)
0501 19:56:52.538929 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:52.538932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:52.538945 (+ 13us) server_negotiation.cc:407] Connection header received
0501 19:56:52.539057 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:52.539060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:52.539105 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:52.539185 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:52.539713 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:52.540239 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:52.540813 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:52.540939 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:52.543171 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:39741: BlockingRecv error: recv got EOF from 127.8.168.65:39741 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":75,"threads_started":1}
May 01 19:56:52 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:52 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:52.710893 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:52.706141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:32949)
0501 19:56:52.706375 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:52.706387 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:52.706484 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:52.706706 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:52.706708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:52.706876 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:52.707119 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:52.707126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:52.708043 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:52.708047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:52.708533 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:52.708543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:52.708634 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:52.709214 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:52.709227 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:52.710783 (+ 1556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":67,"threads_started":1}
W20240501 19:56:52.711017 18061 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:52.706215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:32949 (local address 127.8.168.126:33783)
0501 19:56:52.706368 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:52.706371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:52.706451 (+ 80us) server_negotiation.cc:407] Connection header received
0501 19:56:52.706556 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:52.706559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:52.706611 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:52.706686 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:52.707241 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:52.707924 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:52.708641 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:52.708829 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:52.710911 (+ 2082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:32949: BlockingRecv error: recv got EOF from 127.8.168.66:32949 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
May 01 19:56:52 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:52 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:52.881749 18063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:52.877063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:52191)
0501 19:56:52.877345 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:52.877359 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:52.877475 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:52.877755 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:52.877758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:52.877947 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:52.878181 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:52.878188 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:52.878893 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:52.878896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:52.879292 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:52.879300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:52.879399 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:52.879928 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:52.879941 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:52.881634 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":92,"threads_started":1}
W20240501 19:56:52.881877 18061 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:52.877152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:52191 (local address 127.8.168.126:33783)
0501 19:56:52.877293 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:52.877297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:52.877416 (+ 119us) server_negotiation.cc:407] Connection header received
0501 19:56:52.877578 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:52.877582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:52.877655 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:52.877726 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:52.878297 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:52.878783 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:52.879404 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:52.879580 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:52.881761 (+ 2181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.67:52191: BlockingRecv error: recv got EOF from 127.8.168.67:52191 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
May 01 19:56:53 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:53 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:53.548699 18064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:53.543876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:45615)
0501 19:56:53.544166 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:53.544179 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:53.544267 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:53.544547 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:53.544550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:53.544760 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:53.544959 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:53.544965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:53.545657 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:53.545659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:53.546041 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:53.546047 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:53.546134 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:53.546666 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:53.546680 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:53.548577 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":110,"threads_started":1}
W20240501 19:56:53.548974 18065 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:53.543978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:45615 (local address 127.8.168.126:33783)
0501 19:56:53.544219 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:53.544222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:53.544235 (+ 13us) server_negotiation.cc:407] Connection header received
0501 19:56:53.544345 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:53.544347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:53.544390 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:53.544466 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:53.545056 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:53.545568 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:53.546124 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:53.546293 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:53.548782 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:45615: BlockingRecv error: recv got EOF from 127.8.168.65:45615 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":65,"threads_started":1}
May 01 19:56:53 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:53 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:53.717015 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:53.711735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:41173)
0501 19:56:53.712072 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:53.712088 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:53.712215 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:53.712470 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:53.712473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:53.712673 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:53.712926 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:53.712934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:53.713680 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:53.713682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:53.714113 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:53.714121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:53.714225 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:53.714792 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:53.714809 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:53.716896 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":108,"threads_started":1}
W20240501 19:56:53.717150 18065 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:53.711880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:41173 (local address 127.8.168.126:33783)
0501 19:56:53.712064 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:53.712082 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:53.712175 (+ 93us) server_negotiation.cc:407] Connection header received
0501 19:56:53.712308 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:53.712312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:53.712360 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:53.712431 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:53.713047 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:53.713566 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:53.714218 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:53.714398 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:53.717029 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:41173: BlockingRecv error: recv got EOF from 127.8.168.66:41173 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
May 01 19:56:53 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:53 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:53.887099 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:53.882475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:58595)
0501 19:56:53.882739 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:53.882752 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:53.882853 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:53.883128 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:53.883131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:53.883320 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:53.883517 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:53.883524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:53.884249 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:53.884251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:53.884661 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:53.884669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:53.884780 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:53.885288 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:53.885299 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:53.886952 (+ 1653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":72,"threads_started":1}
W20240501 19:56:53.887207 18065 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:53.882563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:58595 (local address 127.8.168.126:33783)
0501 19:56:53.882726 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:53.882729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:53.882835 (+ 106us) server_negotiation.cc:407] Connection header received
0501 19:56:53.882931 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:53.882933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:53.882981 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:53.883047 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:53.883630 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:53.884154 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:53.884772 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:53.884902 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:53.887107 (+ 2205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.67:58595: BlockingRecv error: recv got EOF from 127.8.168.67:58595 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
May 01 19:56:54 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:54 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:54.554692 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:54.549508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:60555)
0501 19:56:54.549782 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:54.549798 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:54.549896 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:54.550162 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:54.550166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:54.550446 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:54.550726 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:54.550736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:54.551513 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:54.551517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:54.551896 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:54.551904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:54.551996 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:54.552528 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:54.552540 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:54.554562 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
W20240501 19:56:54.554836 18069 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:54.549652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:60555 (local address 127.8.168.126:33783)
0501 19:56:54.549903 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:54.549907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:54.549921 (+ 14us) server_negotiation.cc:407] Connection header received
0501 19:56:54.549975 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:54.549977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:54.550045 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:54.550126 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:54.550858 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:54.551413 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:54.551993 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:54.552133 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:54.554714 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:60555: BlockingRecv error: recv got EOF from 127.8.168.65:60555 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":90,"threads_started":1}
May 01 19:56:54 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:54 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:54.722235 18070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:54.717733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:57315)
0501 19:56:54.718019 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:54.718032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:54.718126 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:54.718345 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:54.718347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:54.718537 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:54.718740 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:54.718745 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:54.719445 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:54.719448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:54.719819 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:54.719825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:54.719908 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:54.720410 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:54.720422 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:54.722073 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":94,"threads_started":1}
W20240501 19:56:54.722340 18069 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:54.717888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:57315 (local address 127.8.168.126:33783)
0501 19:56:54.718020 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:54.718023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:54.718093 (+ 70us) server_negotiation.cc:407] Connection header received
0501 19:56:54.718198 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:54.718200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:54.718242 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:54.718313 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:54.718846 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:54.719358 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:54.719899 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:54.720028 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:54.722222 (+ 2194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:57315: BlockingRecv error: recv got EOF from 127.8.168.66:57315 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
May 01 19:56:54 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:54 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:54.892454 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:54.887913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:58835)
0501 19:56:54.888207 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:54.888219 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:54.888323 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:54.888576 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:54.888579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:54.888819 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:54.889009 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:54.889015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:54.889716 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:54.889719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:54.890085 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:54.890092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:54.890192 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:54.890707 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:54.890720 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:54.892294 (+ 1574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":103,"threads_started":1}
W20240501 19:56:54.892537 18069 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:54.888006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:58835 (local address 127.8.168.126:33783)
0501 19:56:54.888152 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:54.888157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:54.888291 (+ 134us) server_negotiation.cc:407] Connection header received
0501 19:56:54.888412 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:54.888416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:54.888460 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:54.888524 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:54.889131 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:54.889609 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:54.890195 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:54.890367 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:54.892438 (+ 2071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.67:58835: BlockingRecv error: recv got EOF from 127.8.168.67:58835 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
May 01 19:56:55 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:55 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:55.564570 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:55.555507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:56533)
0501 19:56:55.555781 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:55.555795 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:55.555892 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:55.556284 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:55.556287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:55.556475 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:55.556683 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:55.556689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:55.557479 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:55.557482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:55.557867 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:55.557874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:55.557965 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:55.562693 (+ 4728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:55.562708 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:55.564447 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
W20240501 19:56:55.564714 18073 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:55.555602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:56533 (local address 127.8.168.126:33783)
0501 19:56:55.555866 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:55.555870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:55.555885 (+ 15us) server_negotiation.cc:407] Connection header received
0501 19:56:55.556012 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:55.556015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:55.556059 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:55.556259 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:55.556848 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:55.557381 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:55.557955 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:55.558084 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:55.564596 (+ 6512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:56533: BlockingRecv error: recv got EOF from 127.8.168.65:56533 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
May 01 19:56:55 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:55 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:55.727978 18074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:55.722905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:55557)
0501 19:56:55.723204 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:55.723221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:55.723343 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:55.723619 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:55.723623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:55.723823 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:55.724023 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:55.724030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:55.724882 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:55.724886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:55.725296 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:55.725306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:55.725427 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:55.725997 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:55.726011 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:55.727816 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
W20240501 19:56:55.728044 18073 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:55.722992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:55557 (local address 127.8.168.126:33783)
0501 19:56:55.723179 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:55.723183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:55.723299 (+ 116us) server_negotiation.cc:407] Connection header received
0501 19:56:55.723437 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:55.723441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:55.723503 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:55.723582 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:55.724160 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:55.724685 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:55.725425 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:55.725605 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:55.727949 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:55557: BlockingRecv error: recv got EOF from 127.8.168.66:55557 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
May 01 19:56:55 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:55 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:55.897725 18075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:55.893330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:49515)
0501 19:56:55.893575 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:55.893588 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:55.893685 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:55.893930 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:55.893933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:55.894108 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:55.894334 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:55.894342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:55.895022 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:55.895024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:55.895393 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:55.895398 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:55.895478 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:55.895989 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:55.896000 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:55.897618 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
W20240501 19:56:55.897861 18073 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:55.893392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:49515 (local address 127.8.168.126:33783)
0501 19:56:55.893524 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:55.893528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:55.893649 (+ 121us) server_negotiation.cc:407] Connection header received
0501 19:56:55.893755 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:55.893758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:55.893812 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:55.893882 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:55.894439 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:55.894937 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:55.895472 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:55.895591 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:55.897746 (+ 2155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.67:49515: BlockingRecv error: recv got EOF from 127.8.168.67:49515 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
May 01 19:56:56 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:56 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:56.569783 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:56.565312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:47623)
0501 19:56:56.565607 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:56.565620 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:56.565726 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:56.565948 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:56.565950 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:56.566118 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:56.566311 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:56.566316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:56.567001 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:56.567004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:56.567370 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:56.567375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:56.567459 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:56.567988 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:56.568001 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:56.569662 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":71,"threads_started":1}
W20240501 19:56:56.569952 18077 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:56.565379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:47623 (local address 127.8.168.126:33783)
0501 19:56:56.565638 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:56.565642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:56.565678 (+ 36us) server_negotiation.cc:407] Connection header received
0501 19:56:56.565802 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:56.565804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:56.565847 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:56.565922 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:56.566400 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:56.566920 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:56.567446 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:56.567588 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:56.569816 (+ 2228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:47623: BlockingRecv error: recv got EOF from 127.8.168.65:47623 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
May 01 19:56:56 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:56 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:56.733276 18078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:56.728774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:33301)
0501 19:56:56.729063 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:56.729076 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:56.729178 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:56.729400 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:56.729402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:56.729575 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:56.729783 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:56.729789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:56.730474 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:56.730476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:56.730850 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:56.730856 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:56.730939 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:56.731477 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:56.731490 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:56.733152 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":78,"threads_started":1}
W20240501 19:56:56.733386 18077 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:56.728842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:33301 (local address 127.8.168.126:33783)
0501 19:56:56.728987 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:56.728991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:56.729145 (+ 154us) server_negotiation.cc:407] Connection header received
0501 19:56:56.729252 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:56.729255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:56.729302 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:56.729371 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:56.729883 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:56.730389 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:56.730944 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:56.731106 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:56.733283 (+ 2177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:33301: BlockingRecv error: recv got EOF from 127.8.168.66:33301 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
May 01 19:56:56 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:56 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:56.902878 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:56.898432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:54771)
0501 19:56:56.898711 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:56.898723 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:56.898818 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:56.899045 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:56.899047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:56.899211 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:56.899402 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:56.899408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:56.900092 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:56.900094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:56.900464 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:56.900470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:56.900554 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:56.901098 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:56.901111 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:56.902763 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":72,"threads_started":1}
W20240501 19:56:56.903010 18077 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:56.898507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:54771 (local address 127.8.168.126:33783)
0501 19:56:56.898643 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:56.898646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:56.898786 (+ 140us) server_negotiation.cc:407] Connection header received
0501 19:56:56.898903 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:56.898905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:56.898950 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:56.899029 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:56.899499 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:56.899980 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:56.900544 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:56.900669 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:56.902896 (+ 2227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.67:54771: BlockingRecv error: recv got EOF from 127.8.168.67:54771 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
May 01 19:56:57 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:57 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:57.575165 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:57.570562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:53385)
0501 19:56:57.570852 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:57.570867 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:57.570953 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:57.571186 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:57.571188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:57.571360 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:57.571549 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:57.571554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:57.572244 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:57.572246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:57.572613 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:57.572618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:57.572752 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:57.573272 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:57.573285 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:57.575050 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":76,"threads_started":1}
W20240501 19:56:57.575325 18081 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:57.570686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:53385 (local address 127.8.168.126:33783)
0501 19:56:57.570924 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:57.570927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:57.570940 (+ 13us) server_negotiation.cc:407] Connection header received
0501 19:56:57.571037 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:57.571040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:57.571083 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:57.571158 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:57.571639 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:57.572161 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:57.572696 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:57.572866 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:57.575190 (+ 2324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:53385: BlockingRecv error: recv got EOF from 127.8.168.65:53385 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
May 01 19:56:57 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:57 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:57.738932 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:57.734103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:48985)
0501 19:56:57.734407 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:57.734422 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:57.734524 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:57.734826 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:57.734829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:57.735036 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:57.735280 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:57.735290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:57.736168 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:57.736170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:57.736540 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:57.736546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:57.736636 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:57.737202 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:57.737215 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:57.738807 (+ 1592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":100,"threads_started":1}
W20240501 19:56:57.739188 18081 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:57.734211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:48985 (local address 127.8.168.126:33783)
0501 19:56:57.734467 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:57.734472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:57.734483 (+ 11us) server_negotiation.cc:407] Connection header received
0501 19:56:57.734639 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:57.734643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:57.734701 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:57.734776 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:57.735398 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:57.736082 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:57.736632 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:57.736837 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:57.738936 (+ 2099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:48985: BlockingRecv error: recv got EOF from 127.8.168.66:48985 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
May 01 19:56:57 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:57 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:57.908088 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:57.903633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:43015)
0501 19:56:57.903870 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:57.903882 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:57.903984 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:57.904193 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:57.904195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:57.904364 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:57.904553 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:57.904559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:57.905241 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:57.905243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:57.905630 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:57.905636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:57.905712 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:57.906289 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:57.906304 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:57.907960 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":67,"threads_started":1}
W20240501 19:56:57.908179 18081 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:57.903726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:43015 (local address 127.8.168.126:33783)
0501 19:56:57.903879 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:57.903882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:57.903949 (+ 67us) server_negotiation.cc:407] Connection header received
0501 19:56:57.904057 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:57.904059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:57.904103 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:57.904169 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:57.904653 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:57.905167 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:57.905710 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:57.905867 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:57.908084 (+ 2217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.67:43015: BlockingRecv error: recv got EOF from 127.8.168.67:43015 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
May 01 19:56:58 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:58 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:58.580826 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:58.575912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:41435)
0501 19:56:58.576185 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:58.576199 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:58.576290 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:58.576525 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:58.576527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:58.576701 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:58.576951 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:58.576958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:58.577741 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:58.577744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:58.578122 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:58.578130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:58.578236 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:58.578820 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:58.578834 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:58.580668 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":90,"threads_started":1}
W20240501 19:56:58.580956 18085 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:58.576023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:41435 (local address 127.8.168.126:33783)
0501 19:56:58.576242 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:58.576245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:58.576258 (+ 13us) server_negotiation.cc:407] Connection header received
0501 19:56:58.576375 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:58.576377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:58.576421 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:58.576499 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:58.577082 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:58.577623 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:58.578234 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:58.578423 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:58.580828 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:41435: BlockingRecv error: recv got EOF from 127.8.168.65:41435 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":56,"threads_started":1}
May 01 19:56:58 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:58 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:58.744024 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:58.739644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:46803)
0501 19:56:58.739890 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:58.739902 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:58.739999 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:58.740230 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:58.740232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:58.740400 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:58.740591 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:58.740597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:58.741292 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:58.741295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:58.741667 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:58.741674 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:58.741764 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:58.742276 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:58.742289 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:58.743909 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
W20240501 19:56:58.744161 18085 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:58.739732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:46803 (local address 127.8.168.126:33783)
0501 19:56:58.739858 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:58.739862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:58.739988 (+ 126us) server_negotiation.cc:407] Connection header received
0501 19:56:58.740079 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:58.740082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:58.740130 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:58.740205 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:58.740677 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:58.741189 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:58.741752 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:58.741874 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:58.744037 (+ 2163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:46803: BlockingRecv error: recv got EOF from 127.8.168.66:46803 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
May 01 19:56:58 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:58 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:58.913311 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:58.908750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:50491)
0501 19:56:58.909020 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:58.909032 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:58.909140 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:58.909373 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:58.909376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:58.909579 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:58.909766 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:58.909773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:58.910487 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:58.910491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:58.910866 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:58.910873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:58.910972 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:58.911483 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:58.911496 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:58.913198 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":97,"threads_started":1}
W20240501 19:56:58.913429 18085 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:58.908833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:50491 (local address 127.8.168.126:33783)
0501 19:56:58.908977 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:58.908981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:58.909097 (+ 116us) server_negotiation.cc:407] Connection header received
0501 19:56:58.909223 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:58.909227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:58.909274 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:58.909338 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:58.909886 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:58.910386 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:58.910968 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:58.911152 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:58.913317 (+ 2165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.67:50491: BlockingRecv error: recv got EOF from 127.8.168.67:50491 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
May 01 19:56:59 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:59 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:59.586355 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:59.581609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:45171)
0501 19:56:59.581875 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:59.581891 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:59.581991 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:59.582300 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:59.582303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:59.582506 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:59.582696 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:59.582702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:59.583452 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:59.583455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:59.583832 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:59.583839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:59.583939 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:59.584500 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:59.584512 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:59.586209 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
W20240501 19:56:59.586460 18089 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:59.581719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:45171 (local address 127.8.168.126:33783)
0501 19:56:59.581952 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:59.581956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:59.581969 (+ 13us) server_negotiation.cc:407] Connection header received
0501 19:56:59.582123 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:59.582127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:59.582183 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:59.582266 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:59.582813 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:59.583347 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:59.583937 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:59.584122 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:59.586350 (+ 2228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:45171: BlockingRecv error: recv got EOF from 127.8.168.65:45171 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
May 01 19:56:59 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:59 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:59.749280 18090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:59.744761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:39509)
0501 19:56:59.745001 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:59.745014 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:59.745105 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:59.745326 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:59.745329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:59.745498 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:59.745694 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:59.745701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:59.746413 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:59.746416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:59.746800 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:59.746809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:59.746895 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:59.747444 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:59.747458 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:59.749167 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
W20240501 19:56:59.749409 18089 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:59.744797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:39509 (local address 127.8.168.126:33783)
0501 19:56:59.744947 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:59.744951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:59.745076 (+ 125us) server_negotiation.cc:407] Connection header received
0501 19:56:59.745190 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:59.745192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:59.745237 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:59.745307 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:59.745790 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:59.746324 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:59.746902 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:59.747051 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:59.749292 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:39509: BlockingRecv error: recv got EOF from 127.8.168.66:39509 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
May 01 19:56:59 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:56:59 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:56:59.921734 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:56:59.914014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:58245)
0501 19:56:59.914281 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:59.914294 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:59.914396 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:59.914650 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:56:59.914653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:56:59.914830 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:56:59.915055 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:59.915063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:59.915789 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:56:59.915792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:56:59.916191 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:56:59.916199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:56:59.916289 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:59.919948 (+ 3659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:56:59.919961 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:56:59.921619 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
W20240501 19:56:59.921871 18089 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:59.914153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:58245 (local address 127.8.168.126:33783)
0501 19:56:59.914293 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0501 19:56:59.914296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:56:59.914363 (+ 67us) server_negotiation.cc:407] Connection header received
0501 19:56:59.914466 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:56:59.914468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:56:59.914514 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:56:59.914579 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:56:59.915176 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:59.915695 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:56:59.916278 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:56:59.916399 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:56:59.921755 (+ 5356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.67:58245: BlockingRecv error: recv got EOF from 127.8.168.67:58245 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
May 01 19:57:00 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:57:00 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:57:00.594754 18092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:00.587130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:60377)
0501 19:57:00.587375 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:00.587387 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:00.587476 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:00.587710 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:00.587711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:00.587878 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:00.588070 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:00.588076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:00.588766 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:00.588770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:00.589134 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:00.589141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:00.589238 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:00.592891 (+ 3653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:57:00.592905 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:00.594603 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
W20240501 19:57:00.594871 18093 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:00.587192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:60377 (local address 127.8.168.126:33783)
0501 19:57:00.587404 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:00.587407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:00.587433 (+ 26us) server_negotiation.cc:407] Connection header received
0501 19:57:00.587556 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:00.587558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:00.587603 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:00.587689 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:00.588156 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:00.588660 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:00.589226 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:00.589359 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:00.594751 (+ 5392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:60377: BlockingRecv error: recv got EOF from 127.8.168.65:60377 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":64,"threads_started":1}
May 01 19:57:00 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:57:00 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:57:00.755210 18094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:00.749957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:47945)
0501 19:57:00.750244 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:00.750259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:00.750382 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:00.750630 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:00.750633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:00.750852 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:00.751064 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:00.751072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:00.751926 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:00.751930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:00.752390 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:00.752402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:00.752512 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:00.753263 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:57:00.753276 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:00.755085 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":85,"threads_started":1}
W20240501 19:57:00.755376 18093 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:00.750054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:47945 (local address 127.8.168.126:33783)
0501 19:57:00.750230 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:00.750235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:00.750335 (+ 100us) server_negotiation.cc:407] Connection header received
0501 19:57:00.750469 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:00.750472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:00.750525 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:00.750595 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:00.751192 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:00.751792 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:00.752550 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:00.752861 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:00.755254 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:47945: BlockingRecv error: recv got EOF from 127.8.168.66:47945 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
May 01 19:57:00 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:57:00 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:57:00.927330 18095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:00.922471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:34621)
0501 19:57:00.922713 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:00.922726 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:00.922829 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:00.923061 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:00.923063 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:00.923233 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:00.923421 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:00.923427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:00.924202 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:00.924205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:00.924595 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:00.924603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:00.924691 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:00.925375 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:57:00.925392 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:00.927215 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
W20240501 19:57:00.927469 18093 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:00.922556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:34621 (local address 127.8.168.126:33783)
0501 19:57:00.922716 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:00.922719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:00.922793 (+ 74us) server_negotiation.cc:407] Connection header received
0501 19:57:00.922908 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:00.922910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:00.922955 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:00.923027 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:00.923573 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:00.924102 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:00.924703 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:00.924897 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:00.927355 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.67:34621: BlockingRecv error: recv got EOF from 127.8.168.67:34621 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
May 01 19:57:01 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:57:01 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:57:01.600601 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:01.595763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:40431)
0501 19:57:01.596033 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:01.596050 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:01.596152 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:01.596372 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:01.596374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:01.596551 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:01.596787 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:01.596794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:01.597528 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:01.597531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:01.597919 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:01.597927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:01.598032 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:01.598550 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:57:01.598564 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:01.600449 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":67,"threads_started":1}
W20240501 19:57:01.600709 18097 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:01.595874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:40431 (local address 127.8.168.126:33783)
0501 19:57:01.596129 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:01.596132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:01.596145 (+ 13us) server_negotiation.cc:407] Connection header received
0501 19:57:01.596233 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:01.596235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:01.596278 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:01.596345 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:01.596885 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:01.597399 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:01.598007 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:01.598133 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:01.600596 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:40431: BlockingRecv error: recv got EOF from 127.8.168.65:40431 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
May 01 19:57:01 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:57:01 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:57:01.760692 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:01.755953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:35055)
0501 19:57:01.756247 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:01.756259 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:01.756355 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:01.756681 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:01.756684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:01.756873 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:01.757065 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:01.757071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:01.757797 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:01.757800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:01.758183 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:01.758190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:01.758282 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:01.758827 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:57:01.758840 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:01.760574 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":90,"threads_started":1}
W20240501 19:57:01.760891 18097 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:01.756010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:35055 (local address 127.8.168.126:33783)
0501 19:57:01.756149 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:01.756152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:01.756320 (+ 168us) server_negotiation.cc:407] Connection header received
0501 19:57:01.756489 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:01.756492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:01.756543 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:01.756623 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:01.757176 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:01.757685 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:01.758292 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:01.758426 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:01.760714 (+ 2288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:35055: BlockingRecv error: recv got EOF from 127.8.168.66:35055 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
May 01 19:57:01 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:57:01 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:57:01.932461 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:01.928001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:60823)
0501 19:57:01.928253 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:01.928265 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:01.928359 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:01.928565 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:01.928567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:01.928758 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:01.928950 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:01.928956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:01.929616 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:01.929619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:01.929989 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:01.929994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:01.930073 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:01.930598 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:57:01.930610 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:01.932314 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":70,"threads_started":1}
W20240501 19:57:01.932554 18097 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:01.928093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:60823 (local address 127.8.168.126:33783)
0501 19:57:01.928219 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:01.928222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:01.928323 (+ 101us) server_negotiation.cc:407] Connection header received
0501 19:57:01.928429 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:01.928431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:01.928473 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:01.928540 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:01.929052 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:01.929534 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:01.930084 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:01.930207 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:01.932453 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.67:60823: BlockingRecv error: recv got EOF from 127.8.168.67:60823 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
May 01 19:57:02 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:57:02 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:57:02.606148 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:02.601385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:59713)
0501 19:57:02.601652 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:02.601673 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:02.601771 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:02.602001 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:02.602003 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:02.602172 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:02.602380 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:02.602386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:02.603214 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:02.603218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:02.603703 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:02.603713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:02.603806 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:02.604309 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:57:02.604320 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:02.605993 (+ 1673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
W20240501 19:57:02.606245 18101 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:02.601456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:59713 (local address 127.8.168.126:33783)
0501 19:57:02.601673 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:02.601676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:02.601722 (+ 46us) server_negotiation.cc:407] Connection header received
0501 19:57:02.601847 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:02.601849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:02.601894 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:02.601977 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:02.602494 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:02.603087 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:02.603801 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:02.603939 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:02.606132 (+ 2193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:59713: BlockingRecv error: recv got EOF from 127.8.168.65:59713 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":67,"threads_started":1}
May 01 19:57:02 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:57:02 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:57:02.766695 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:02.761468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:37533)
0501 19:57:02.761777 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:02.761792 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:02.761905 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:02.762175 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:02.762178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:02.762355 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:02.762559 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:02.762565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:02.763359 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:02.763363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:02.763865 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:02.763877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:02.764011 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:02.764677 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:57:02.764693 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:02.766558 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":87,"threads_started":1}
W20240501 19:57:02.766804 18101 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:02.761665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:37533 (local address 127.8.168.126:33783)
0501 19:57:02.761808 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:02.761812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:02.761848 (+ 36us) server_negotiation.cc:407] Connection header received
0501 19:57:02.761985 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:02.761988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:02.762033 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:02.762116 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:02.762666 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:02.763238 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:02.763999 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:02.764184 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:02.766703 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:37533: BlockingRecv error: recv got EOF from 127.8.168.66:37533 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
May 01 19:57:02 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:57:02 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:57:02.937932 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:02.933256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:37791)
0501 19:57:02.933501 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:02.933513 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:02.933628 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:02.933893 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:02.933895 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:02.934062 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:02.934253 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:02.934258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:02.935002 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:02.935006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:02.935505 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:02.935518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:02.935654 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:02.936177 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:57:02.936190 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:02.937820 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
W20240501 19:57:02.938056 18101 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:02.933323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:37791 (local address 127.8.168.126:33783)
0501 19:57:02.933455 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:02.933458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:02.933594 (+ 136us) server_negotiation.cc:407] Connection header received
0501 19:57:02.933737 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:02.933740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:02.933798 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:02.933866 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:02.934344 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:02.934897 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:02.935643 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:02.935833 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:02.937940 (+ 2107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.67:37791: BlockingRecv error: recv got EOF from 127.8.168.67:37791 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
May 01 19:57:03 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:57:03 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:57:03.611543 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:03.606842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:54003)
0501 19:57:03.607127 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:03.607143 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:03.607249 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:03.607493 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:03.607495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:03.607665 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:03.607867 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:03.607872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:03.608552 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:03.608554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:03.608974 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:03.608982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:03.609078 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:03.609612 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:57:03.609625 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:03.611406 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
W20240501 19:57:03.611670 18105 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:03.607013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:54003 (local address 127.8.168.126:33783)
0501 19:57:03.607298 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:03.607301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:03.607313 (+ 12us) server_negotiation.cc:407] Connection header received
0501 19:57:03.607344 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:03.607346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:03.607389 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:03.607462 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:03.607956 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:03.608471 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:03.609057 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:03.609188 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:03.611542 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:54003: BlockingRecv error: recv got EOF from 127.8.168.65:54003 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":78,"threads_started":1}
May 01 19:57:03 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:57:03 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:57:03.772029 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:03.767413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:33841)
0501 19:57:03.767673 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:03.767688 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:03.767798 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:03.768031 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:03.768034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:03.768238 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:03.768431 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:03.768437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:03.769207 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:03.769210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:03.769579 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:03.769587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:03.769688 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:03.770235 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:57:03.770249 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:03.771913 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":87,"threads_started":1}
W20240501 19:57:03.772179 18105 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:03.767501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:33841 (local address 127.8.168.126:33783)
0501 19:57:03.767669 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:03.767674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:03.767755 (+ 81us) server_negotiation.cc:407] Connection header received
0501 19:57:03.767880 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:03.767883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:03.767929 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:03.767995 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:03.768548 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:03.769099 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:03.769690 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:03.769861 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:03.772047 (+ 2186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:33841: BlockingRecv error: recv got EOF from 127.8.168.66:33841 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
May 01 19:57:03 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:57:03 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.67@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:57:03.943353 18107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:03.938672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.67:47995)
0501 19:57:03.939034 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:03.939046 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:03.939148 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:03.939383 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:03.939385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:03.939571 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:03.939768 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:03.939775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:03.940477 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:03.940480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:03.940893 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:03.940901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:03.941002 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:03.941535 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:57:03.941547 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:03.943238 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":114,"threads_started":1}
W20240501 19:57:03.943467 18105 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:03.938734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:47995 (local address 127.8.168.126:33783)
0501 19:57:03.938871 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:03.938874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:03.939117 (+ 243us) server_negotiation.cc:407] Connection header received
0501 19:57:03.939230 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:03.939233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:03.939280 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:03.939344 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:03.939885 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:03.940374 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:03.941003 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:03.941196 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:03.943364 (+ 2168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.67:47995: BlockingRecv error: recv got EOF from 127.8.168.67:47995 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
May 01 19:57:04 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:57:04 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.65@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:57:04.617290 18108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:04.612467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.65:58025)
0501 19:57:04.612768 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:04.612782 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:04.612870 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:04.613082 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:04.613084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:04.613267 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:04.613457 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:04.613462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:04.614245 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:04.614248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:04.614624 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:04.614631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:04.614716 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:04.615317 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:57:04.615329 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:04.617161 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
W20240501 19:57:04.617429 18109 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:04.612566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:58025 (local address 127.8.168.126:33783)
0501 19:57:04.612864 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:04.612868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:04.612882 (+ 14us) server_negotiation.cc:407] Connection header received
0501 19:57:04.612937 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:04.612939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:04.612981 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:04.613060 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:04.613545 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:04.614133 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:04.614724 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:04.614874 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:04.617298 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.65:58025: BlockingRecv error: recv got EOF from 127.8.168.65:58025 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":92,"threads_started":1}
May 01 19:57:04 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
May 01 19:57:04 dist-test-slave-tzct krb5kdc[17050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.168.66@KRBTEST.COM for oryx/127.8.168.126@KRBTEST.COM, Server not found in Kerberos database
I20240501 19:57:04.777590 18110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:04.772887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33783 (local address 127.8.168.66:51977)
0501 19:57:04.773126 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:04.773139 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:04.773235 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:04.773449 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:04.773451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:04.773621 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:04.773808 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:04.773814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:04.774682 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:04.774686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:04.775108 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:04.775119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:04.775242 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:04.775833 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:57:04.775847 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:04.777469 (+ 1622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.168.126:33783: Server oryx/127.8.168.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":70,"threads_started":1}
W20240501 19:57:04.777709 18109 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:04.772978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:51977 (local address 127.8.168.126:33783)
0501 19:57:04.773149 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:04.773152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:04.773193 (+ 41us) server_negotiation.cc:407] Connection header received
0501 19:57:04.773306 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:04.773309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:04.773355 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:04.773430 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:04.773970 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:04.774539 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:04.775220 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:04.775412 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:04.777595 (+ 2183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.168.66:51977: BlockingRecv error: recv got EOF from 127.8.168.66:51977 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20240501 19:57:04.853751 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 17633
I20240501 19:57:04.858269 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 17786
I20240501 19:57:04.862516 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 17918
I20240501 19:57:04.866727 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 17563
2024-05-01T19:57:04Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17693 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2024-05-01T19:57:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:57:04Z Disabled control of system clock
I20240501 19:57:04.892614 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:38535
--webserver_interface=127.8.168.126
--webserver_port=0
--builtin_ntp_servers=127.8.168.84:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:38535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20240501 19:57:04.994758 18114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:04.994983 18114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:04.995030 18114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:04.998052 18114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:57:04.998109 18114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:57:04.998124 18114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:04.998142 18114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:57:04.998157 18114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:57:05.001689 18114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:38535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:38535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:05.002485 18114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:05.003516 18114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:57:05.011082 18114 server_base.cc:1034] running on GCE node
W20240501 19:57:05.011634 18122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:05.012094 18119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:05.012164 18120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:05.012665 18114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:05.013561 18114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:05.014722 18114 hybrid_clock.cc:648] HybridClock initialized: now 1714593425014706 us; error 29 us; skew 500 ppm
I20240501 19:57:05.016507 18114 webserver.cc:460] Webserver started at http://127.8.168.126:33949/ using document root <none> and password file <none>
I20240501 19:57:05.017068 18114 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:05.017125 18114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:05.017303 18114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:05.019033 18114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "b0f8b06708004b8192f18261aaf0b123"
format_stamp: "Formatted at 2024-05-01 19:57:05 on dist-test-slave-tzct"
server_key: "eb5e2328e8119fc816f8aa4727bc468b"
server_key_iv: "646336d9fec474896a74764739cd6793"
server_key_version: "encryptionkey@0"
I20240501 19:57:05.019512 18114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "b0f8b06708004b8192f18261aaf0b123"
format_stamp: "Formatted at 2024-05-01 19:57:05 on dist-test-slave-tzct"
server_key: "eb5e2328e8119fc816f8aa4727bc468b"
server_key_iv: "646336d9fec474896a74764739cd6793"
server_key_version: "encryptionkey@0"
I20240501 19:57:05.022665 18114 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:57:05.024758 18128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:05.025710 18114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:05.025817 18114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "b0f8b06708004b8192f18261aaf0b123"
format_stamp: "Formatted at 2024-05-01 19:57:05 on dist-test-slave-tzct"
server_key: "eb5e2328e8119fc816f8aa4727bc468b"
server_key_iv: "646336d9fec474896a74764739cd6793"
server_key_version: "encryptionkey@0"
I20240501 19:57:05.025902 18114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:05.046487 18114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:05.047151 18114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:05.047302 18114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:05.053591 18114 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:38535
I20240501 19:57:05.053601 18180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:38535 every 8 connection(s)
I20240501 19:57:05.054538 18114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:57:05.057008 18181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.057400 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 18114
I20240501 19:57:05.057494 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:57:05.057714 8865 external_mini_cluster.cc:1428] Setting key c1740902c23bb5e23cd2806d0d966ca1
I20240501 19:57:05.063580 18181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123: Bootstrap starting.
I20240501 19:57:05.065668 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:05.058898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.8.168.126:38535)
0501 19:57:05.059553 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:05.059576 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:05.059618 (+ 42us) server_negotiation.cc:407] Connection header received
0501 19:57:05.060224 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:05.060248 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:05.060679 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:05.061077 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:05.061524 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:05.062625 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:05.063236 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:05.063496 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:05.063934 (+ 438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:57:05.063997 (+ 63us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:57:05.064009 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:57:05.064434 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:57:05.064486 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:57:05.064506 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:57:05.064634 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:05.064761 (+ 127us) server_negotiation.cc:299] Negotiation successful
0501 19:57:05.064991 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":102,"threads_started":1}
I20240501 19:57:05.065948 18181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:05.066715 18181 log.cc:824] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:05.068622 18181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123: No bootstrap required, opened a new log
I20240501 19:57:05.071712 18181 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f8b06708004b8192f18261aaf0b123" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 38535 } }
I20240501 19:57:05.071909 18181 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:05.071951 18181 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f8b06708004b8192f18261aaf0b123, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.072374 18181 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f8b06708004b8192f18261aaf0b123" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 38535 } }
I20240501 19:57:05.072495 18181 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:57:05.072559 18181 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:57:05.072669 18181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:05.073561 18181 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f8b06708004b8192f18261aaf0b123" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 38535 } }
I20240501 19:57:05.073877 18181 leader_election.cc:302] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0f8b06708004b8192f18261aaf0b123; no voters:
I20240501 19:57:05.074169 18181 leader_election.cc:288] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:57:05.074268 18186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:05.074460 18186 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [term 1 LEADER]: Becoming Leader. State: Replica: b0f8b06708004b8192f18261aaf0b123, State: Running, Role: LEADER
I20240501 19:57:05.074738 18186 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f8b06708004b8192f18261aaf0b123" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 38535 } }
I20240501 19:57:05.075248 18181 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:57:05.075881 18187 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0f8b06708004b8192f18261aaf0b123. Latest consensus state: current_term: 1 leader_uuid: "b0f8b06708004b8192f18261aaf0b123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f8b06708004b8192f18261aaf0b123" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 38535 } } }
I20240501 19:57:05.076083 18187 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:05.076207 18188 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0f8b06708004b8192f18261aaf0b123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0f8b06708004b8192f18261aaf0b123" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 38535 } } }
I20240501 19:57:05.076292 18188 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:05.076511 18194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:57:05.078995 18194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:57:05.083549 18194 catalog_manager.cc:1347] Generated new cluster ID: d2156526726949d7bb8e6def1615b8d5
I20240501 19:57:05.083622 18194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:57:05.100109 18194 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:57:05.101014 18194 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:57:05.105423 18194 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b0f8b06708004b8192f18261aaf0b123: Generated new TSK 0
I20240501 19:57:05.106102 18194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:57:05.117703 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:38535
--builtin_ntp_servers=127.8.168.84:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20240501 19:57:05.222841 18205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:05.223032 18205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:05.223081 18205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:05.226197 18205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:57:05.226258 18205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:05.226330 18205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:57:05.230163 18205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:38535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:05.231042 18205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:05.232121 18205 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:05.240144 18213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:05.240348 18210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:05.240854 18211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:05.241376 18205 server_base.cc:1034] running on GCE node
I20240501 19:57:05.241710 18205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:05.242245 18205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:05.243388 18205 hybrid_clock.cc:648] HybridClock initialized: now 1714593425243368 us; error 35 us; skew 500 ppm
I20240501 19:57:05.245087 18205 webserver.cc:460] Webserver started at http://127.8.168.65:40665/ using document root <none> and password file <none>
I20240501 19:57:05.245613 18205 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:05.245671 18205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:05.245841 18205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:05.247550 18205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "6def82bdfd0e48da88025d379d261e62"
format_stamp: "Formatted at 2024-05-01 19:57:05 on dist-test-slave-tzct"
server_key: "f839876703a9862ffbf1843841f7341d"
server_key_iv: "1fd6e59a8c4f2606fc4d9de74a287025"
server_key_version: "encryptionkey@0"
I20240501 19:57:05.247993 18205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "6def82bdfd0e48da88025d379d261e62"
format_stamp: "Formatted at 2024-05-01 19:57:05 on dist-test-slave-tzct"
server_key: "f839876703a9862ffbf1843841f7341d"
server_key_iv: "1fd6e59a8c4f2606fc4d9de74a287025"
server_key_version: "encryptionkey@0"
I20240501 19:57:05.251338 18205 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:57:05.253453 18219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:05.254473 18205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:05.254582 18205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "6def82bdfd0e48da88025d379d261e62"
format_stamp: "Formatted at 2024-05-01 19:57:05 on dist-test-slave-tzct"
server_key: "f839876703a9862ffbf1843841f7341d"
server_key_iv: "1fd6e59a8c4f2606fc4d9de74a287025"
server_key_version: "encryptionkey@0"
I20240501 19:57:05.254664 18205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:05.317657 18205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:05.318339 18205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:05.318503 18205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:05.319059 18205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:05.319953 18205 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:05.320003 18205 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:05.320055 18205 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:05.320084 18205 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:05.329303 18205 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:33271
I20240501 19:57:05.329346 18332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:33271 every 8 connection(s)
I20240501 19:57:05.330264 18205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 19:57:05.333745 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 18205
I20240501 19:57:05.333858 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 19:57:05.334101 8865 external_mini_cluster.cc:1428] Setting key d213ad4d2983ac05d1dbae126bdd1e37
I20240501 19:57:05.336143 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:38535
--builtin_ntp_servers=127.8.168.84:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20240501 19:57:05.338258 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:05.331896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:40873 (local address 127.8.168.126:38535)
0501 19:57:05.332038 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:05.332043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:05.332994 (+ 951us) server_negotiation.cc:407] Connection header received
0501 19:57:05.333789 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:05.333792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:05.333850 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:05.333925 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:05.335449 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:05.336157 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:05.336815 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:05.336995 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:05.337771 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:57:05.337788 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:57:05.337790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:57:05.337802 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:57:05.337821 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:57:05.337829 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:57:05.337896 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:05.338058 (+ 162us) server_negotiation.cc:299] Negotiation successful
0501 19:57:05.338120 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240501 19:57:05.338994 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:05.332205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:38535 (local address 127.8.168.65:40873)
0501 19:57:05.332868 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:05.332902 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:05.333586 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:05.334101 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:05.334109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:05.334187 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:05.335305 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:05.335324 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:05.336293 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:05.336298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:05.336690 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:05.336699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:05.336881 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:05.337510 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 19:57:05.337513 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:05.337557 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 19:57:05.337629 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:57:05.337918 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:57:05.337926 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:57:05.337931 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 19:57:05.338051 (+ 120us) client_negotiation.cc:240] Negotiation successful
0501 19:57:05.338304 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":92,"threads_started":1}
I20240501 19:57:05.339988 18333 heartbeater.cc:347] Connected to a master server at 127.8.168.126:38535
I20240501 19:57:05.340198 18333 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:05.340703 18333 heartbeater.cc:511] Master 127.8.168.126:38535 requested a full tablet report, sending...
I20240501 19:57:05.342036 18145 ts_manager.cc:194] Registered new tserver with Master: 6def82bdfd0e48da88025d379d261e62 (127.8.168.65:33271)
I20240501 19:57:05.343158 18145 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.168.65:40873
W20240501 19:57:05.439893 18336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:05.440133 18336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:05.440194 18336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:05.443302 18336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:57:05.443356 18336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:05.443439 18336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:57:05.446970 18336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:38535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:05.447777 18336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:05.448870 18336 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:05.456894 18341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:05.457365 18344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:05.457573 18342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:05.457988 18336 server_base.cc:1034] running on GCE node
I20240501 19:57:05.458344 18336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:05.458887 18336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:05.460027 18336 hybrid_clock.cc:648] HybridClock initialized: now 1714593425460015 us; error 27 us; skew 500 ppm
I20240501 19:57:05.461697 18336 webserver.cc:460] Webserver started at http://127.8.168.66:37297/ using document root <none> and password file <none>
I20240501 19:57:05.462211 18336 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:05.462267 18336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:05.462430 18336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:05.464149 18336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "81bb601c80bf4059a2019b260229d49c"
format_stamp: "Formatted at 2024-05-01 19:57:05 on dist-test-slave-tzct"
server_key: "c387077ef058891e4f1ca791bf6b2674"
server_key_iv: "c59dd4ec773028d67a79841625841e99"
server_key_version: "encryptionkey@0"
I20240501 19:57:05.464605 18336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "81bb601c80bf4059a2019b260229d49c"
format_stamp: "Formatted at 2024-05-01 19:57:05 on dist-test-slave-tzct"
server_key: "c387077ef058891e4f1ca791bf6b2674"
server_key_iv: "c59dd4ec773028d67a79841625841e99"
server_key_version: "encryptionkey@0"
I20240501 19:57:05.467749 18336 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 19:57:05.469852 18350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:05.470796 18336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:05.470897 18336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "81bb601c80bf4059a2019b260229d49c"
format_stamp: "Formatted at 2024-05-01 19:57:05 on dist-test-slave-tzct"
server_key: "c387077ef058891e4f1ca791bf6b2674"
server_key_iv: "c59dd4ec773028d67a79841625841e99"
server_key_version: "encryptionkey@0"
I20240501 19:57:05.470976 18336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:05.489996 18336 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:05.490636 18336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:05.490792 18336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:05.491345 18336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:05.492224 18336 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:05.492271 18336 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:05.492321 18336 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:05.492348 18336 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:05.500648 18336 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:44617
I20240501 19:57:05.500679 18463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:44617 every 8 connection(s)
I20240501 19:57:05.501544 18336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 19:57:05.502144 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 18336
I20240501 19:57:05.502265 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 19:57:05.502506 8865 external_mini_cluster.cc:1428] Setting key e9ad2d54da72a33465368dbb95410c5e
I20240501 19:57:05.504532 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:38535
--builtin_ntp_servers=127.8.168.84:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20240501 19:57:05.509743 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:05.503123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:46377 (local address 127.8.168.126:38535)
0501 19:57:05.503784 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:05.503788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:05.504452 (+ 664us) server_negotiation.cc:407] Connection header received
0501 19:57:05.505257 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:05.505261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:05.505315 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:05.505405 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:05.506994 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:05.507483 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:05.508287 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:05.508412 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:05.509297 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:57:05.509316 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:57:05.509318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:57:05.509329 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:57:05.509349 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:57:05.509357 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:57:05.509413 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:05.509572 (+ 159us) server_negotiation.cc:299] Negotiation successful
0501 19:57:05.509619 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565}
I20240501 19:57:05.510669 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:05.503512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:38535 (local address 127.8.168.66:46377)
0501 19:57:05.504329 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:05.504362 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:05.505057 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:05.505539 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:05.505547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:05.505628 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:05.506857 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:05.506876 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:05.507610 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:05.507614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:05.508182 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:05.508193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:05.508390 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:05.509016 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 19:57:05.509018 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:05.509068 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 19:57:05.509151 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:57:05.509432 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:57:05.509439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:57:05.509444 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 19:57:05.509578 (+ 134us) client_negotiation.cc:240] Negotiation successful
0501 19:57:05.509908 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":127,"threads_started":1}
I20240501 19:57:05.511699 18464 heartbeater.cc:347] Connected to a master server at 127.8.168.126:38535
I20240501 19:57:05.511911 18464 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:05.512409 18464 heartbeater.cc:511] Master 127.8.168.126:38535 requested a full tablet report, sending...
I20240501 19:57:05.513408 18145 ts_manager.cc:194] Registered new tserver with Master: 81bb601c80bf4059a2019b260229d49c (127.8.168.66:44617)
I20240501 19:57:05.513942 18145 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.168.66:46377
W20240501 19:57:05.612603 18467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:05.612859 18467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:05.612923 18467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:05.615913 18467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:57:05.615972 18467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:05.616075 18467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:57:05.619696 18467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:38535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:05.620646 18467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:05.621750 18467 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:05.629171 18472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:05.629650 18473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:05.630273 18475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:05.630800 18467 server_base.cc:1034] running on GCE node
I20240501 19:57:05.631177 18467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:05.631726 18467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:05.632905 18467 hybrid_clock.cc:648] HybridClock initialized: now 1714593425632888 us; error 27 us; skew 500 ppm
I20240501 19:57:05.637977 18467 webserver.cc:460] Webserver started at http://127.8.168.67:39239/ using document root <none> and password file <none>
I20240501 19:57:05.638554 18467 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:05.638614 18467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:05.638782 18467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:05.640540 18467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "84f62522250e4711b502278de8463dcf"
format_stamp: "Formatted at 2024-05-01 19:57:05 on dist-test-slave-tzct"
server_key: "3f31245f3e247f0f135bfbdd56c63184"
server_key_iv: "5a8a41b15e0ee6552a9484c595cd2d9c"
server_key_version: "encryptionkey@0"
I20240501 19:57:05.641026 18467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "84f62522250e4711b502278de8463dcf"
format_stamp: "Formatted at 2024-05-01 19:57:05 on dist-test-slave-tzct"
server_key: "3f31245f3e247f0f135bfbdd56c63184"
server_key_iv: "5a8a41b15e0ee6552a9484c595cd2d9c"
server_key_version: "encryptionkey@0"
I20240501 19:57:05.644213 18467 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:05.646411 18481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:05.647358 18467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:05.647467 18467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "84f62522250e4711b502278de8463dcf"
format_stamp: "Formatted at 2024-05-01 19:57:05 on dist-test-slave-tzct"
server_key: "3f31245f3e247f0f135bfbdd56c63184"
server_key_iv: "5a8a41b15e0ee6552a9484c595cd2d9c"
server_key_version: "encryptionkey@0"
I20240501 19:57:05.647547 18467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:05.664614 18467 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:05.665264 18467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:05.665409 18467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:05.665921 18467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:05.666769 18467 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:05.666814 18467 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:05.666864 18467 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:05.666889 18467 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:05.674995 18467 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:36823
I20240501 19:57:05.675040 18594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:36823 every 8 connection(s)
I20240501 19:57:05.675891 18467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 19:57:05.679690 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 18467
I20240501 19:57:05.679778 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 19:57:05.679980 8865 external_mini_cluster.cc:1428] Setting key 151b0e75140e55253971d1f77cec1bae
I20240501 19:57:05.683374 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:05.677518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:53665 (local address 127.8.168.126:38535)
0501 19:57:05.677660 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:05.677664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:05.678576 (+ 912us) server_negotiation.cc:407] Connection header received
0501 19:57:05.679294 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:05.679297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:05.679342 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:05.679416 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:05.680797 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:05.681414 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:05.682053 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:05.682226 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:05.682936 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:57:05.682950 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:57:05.682952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:57:05.682963 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:57:05.682980 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:57:05.682987 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:57:05.683045 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:05.683184 (+ 139us) server_negotiation.cc:299] Negotiation successful
0501 19:57:05.683236 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 19:57:05.683967 18597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:05.677815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:38535 (local address 127.8.168.67:53665)
0501 19:57:05.678443 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:05.678476 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:05.679128 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:05.679549 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:05.679556 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:05.679628 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:05.680653 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:05.680669 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:05.681527 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:05.681530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:05.681946 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:05.681954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:05.682132 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:05.682725 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 19:57:05.682726 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:05.682758 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 19:57:05.682810 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:57:05.683055 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:57:05.683062 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:57:05.683068 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 19:57:05.683187 (+ 119us) client_negotiation.cc:240] Negotiation successful
0501 19:57:05.683407 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":98,"threads_started":1}
I20240501 19:57:05.684913 18595 heartbeater.cc:347] Connected to a master server at 127.8.168.126:38535
I20240501 19:57:05.685094 18595 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:05.685552 18595 heartbeater.cc:511] Master 127.8.168.126:38535 requested a full tablet report, sending...
I20240501 19:57:05.686424 18145 ts_manager.cc:194] Registered new tserver with Master: 84f62522250e4711b502278de8463dcf (127.8.168.67:36823)
I20240501 19:57:05.686926 18145 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.168.67:53665
I20240501 19:57:05.693403 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240501 19:57:05.695684 18184 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:05.695015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.8.168.126:38535)
0501 19:57:05.695142 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:05.695146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:05.695240 (+ 94us) server_negotiation.cc:407] Connection header received
0501 19:57:05.695339 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:05.695341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:05.695377 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:05.695440 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:05.695597 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46366: BlockingRecv error: recv got EOF from 127.0.0.1:46366 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20240501 19:57:05.695875 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 18205
I20240501 19:57:05.700438 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 18336
I20240501 19:57:05.704763 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 18467
I20240501 19:57:05.708772 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 18114
2024-05-01T19:57:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (840 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2024-05-01T19:57:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:57:05Z Disabled control of system clock
I20240501 19:57:05.733701 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:40025
--webserver_interface=127.8.168.126
--webserver_port=0
--builtin_ntp_servers=127.8.168.84:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:40025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240501 19:57:05.836489 18606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:05.836755 18606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:05.836815 18606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:05.839753 18606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:57:05.839804 18606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:57:05.839820 18606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:05.839836 18606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:57:05.839851 18606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:57:05.843366 18606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:40025
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:40025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:05.844174 18606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:05.845255 18606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:05.852455 18611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:05.852924 18614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:05.853380 18612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:05.853912 18606 server_base.cc:1034] running on GCE node
I20240501 19:57:05.854568 18606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:05.855643 18606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:05.856781 18606 hybrid_clock.cc:648] HybridClock initialized: now 1714593425856768 us; error 26 us; skew 500 ppm
I20240501 19:57:05.858688 18606 webserver.cc:460] Webserver started at http://127.8.168.126:39731/ using document root <none> and password file <none>
I20240501 19:57:05.859233 18606 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:05.859287 18606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:05.859485 18606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:05.861303 18606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "3029c7f39a5141418251447646e9edb3"
format_stamp: "Formatted at 2024-05-01 19:57:05 on dist-test-slave-tzct"
server_key: "03fc3853de2861bd863792b8ccaab529"
server_key_iv: "6e8041522731d147dda58ec6539423ce"
server_key_version: "encryptionkey@0"
I20240501 19:57:05.861766 18606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "3029c7f39a5141418251447646e9edb3"
format_stamp: "Formatted at 2024-05-01 19:57:05 on dist-test-slave-tzct"
server_key: "03fc3853de2861bd863792b8ccaab529"
server_key_iv: "6e8041522731d147dda58ec6539423ce"
server_key_version: "encryptionkey@0"
I20240501 19:57:05.864972 18606 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:57:05.867121 18620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:05.868137 18606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:05.868253 18606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "3029c7f39a5141418251447646e9edb3"
format_stamp: "Formatted at 2024-05-01 19:57:05 on dist-test-slave-tzct"
server_key: "03fc3853de2861bd863792b8ccaab529"
server_key_iv: "6e8041522731d147dda58ec6539423ce"
server_key_version: "encryptionkey@0"
I20240501 19:57:05.868340 18606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:05.873034 18606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:05.873610 18606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:05.873754 18606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:05.879813 18606 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:40025
I20240501 19:57:05.879822 18672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:40025 every 8 connection(s)
I20240501 19:57:05.880745 18606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:57:05.883163 18673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.888165 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 18606
I20240501 19:57:05.888303 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:57:05.888535 18673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3: Bootstrap starting.
I20240501 19:57:05.888576 8865 external_mini_cluster.cc:1428] Setting key 29d61279f4024b97ac1db892e6809f03
I20240501 19:57:05.890798 18673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:05.891458 18673 log.cc:824] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:05.893321 18673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3: No bootstrap required, opened a new log
I20240501 19:57:05.893810 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:05.889902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57316 (local address 127.8.168.126:40025)
0501 19:57:05.890518 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:05.890527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:05.890563 (+ 36us) server_negotiation.cc:407] Connection header received
0501 19:57:05.891176 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:05.891197 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:05.891489 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:05.891848 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:05.892553 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:57:05.892559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:57:05.892578 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:57:05.892589 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:57:05.892645 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:57:05.892660 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:57:05.892820 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:05.892892 (+ 72us) server_negotiation.cc:299] Negotiation successful
0501 19:57:05.893111 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":130,"threads_started":1}
I20240501 19:57:05.896131 18673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3029c7f39a5141418251447646e9edb3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40025 } }
I20240501 19:57:05.896313 18673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:05.896396 18673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3029c7f39a5141418251447646e9edb3, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.896806 18673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3029c7f39a5141418251447646e9edb3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40025 } }
I20240501 19:57:05.896921 18673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:57:05.896984 18673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:57:05.897085 18673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:05.897837 18673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3029c7f39a5141418251447646e9edb3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40025 } }
I20240501 19:57:05.898134 18673 leader_election.cc:302] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3029c7f39a5141418251447646e9edb3; no voters:
I20240501 19:57:05.898381 18673 leader_election.cc:288] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:57:05.898741 18678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:05.898965 18678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [term 1 LEADER]: Becoming Leader. State: Replica: 3029c7f39a5141418251447646e9edb3, State: Running, Role: LEADER
I20240501 19:57:05.899242 18678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3029c7f39a5141418251447646e9edb3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40025 } }
I20240501 19:57:05.899422 18673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:57:05.900650 18680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3029c7f39a5141418251447646e9edb3. Latest consensus state: current_term: 1 leader_uuid: "3029c7f39a5141418251447646e9edb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3029c7f39a5141418251447646e9edb3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40025 } } }
I20240501 19:57:05.900786 18680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:05.900854 18679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3029c7f39a5141418251447646e9edb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3029c7f39a5141418251447646e9edb3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40025 } } }
I20240501 19:57:05.900960 18679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:05.901317 18687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:57:05.904070 18687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:57:05.908474 18687 catalog_manager.cc:1347] Generated new cluster ID: af1461201fe14c5ea1b1417a0689b1ec
I20240501 19:57:05.908551 18687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:57:05.915969 18687 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:57:05.916282 18687 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:57:05.930445 18687 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3029c7f39a5141418251447646e9edb3: Generated new TSK 0
I20240501 19:57:05.931264 18687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:57:05.937033 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:40025
--builtin_ntp_servers=127.8.168.84:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240501 19:57:06.041639 18697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:06.041841 18697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:06.041908 18697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:06.044862 18697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:57:06.044914 18697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:06.045007 18697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:57:06.048496 18697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:40025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:06.049360 18697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:06.050438 18697 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:06.058277 18702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:06.058574 18703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:06.059147 18705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:06.059590 18697 server_base.cc:1034] running on GCE node
I20240501 19:57:06.059967 18697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:06.060506 18697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:06.061642 18697 hybrid_clock.cc:648] HybridClock initialized: now 1714593426061616 us; error 47 us; skew 500 ppm
I20240501 19:57:06.063457 18697 webserver.cc:460] Webserver started at http://127.8.168.65:44441/ using document root <none> and password file <none>
I20240501 19:57:06.063971 18697 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:06.064028 18697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:06.064201 18697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:06.065924 18697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "1aebc4b783634c8fb9d6701869c2b71c"
format_stamp: "Formatted at 2024-05-01 19:57:06 on dist-test-slave-tzct"
server_key: "bcf6e03241c996ebadaa575d3fcb599a"
server_key_iv: "5c9f6bd36e3cf580ddabef121d2843eb"
server_key_version: "encryptionkey@0"
I20240501 19:57:06.066386 18697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "1aebc4b783634c8fb9d6701869c2b71c"
format_stamp: "Formatted at 2024-05-01 19:57:06 on dist-test-slave-tzct"
server_key: "bcf6e03241c996ebadaa575d3fcb599a"
server_key_iv: "5c9f6bd36e3cf580ddabef121d2843eb"
server_key_version: "encryptionkey@0"
I20240501 19:57:06.069561 18697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:57:06.071655 18711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:06.072678 18697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:06.072808 18697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "1aebc4b783634c8fb9d6701869c2b71c"
format_stamp: "Formatted at 2024-05-01 19:57:06 on dist-test-slave-tzct"
server_key: "bcf6e03241c996ebadaa575d3fcb599a"
server_key_iv: "5c9f6bd36e3cf580ddabef121d2843eb"
server_key_version: "encryptionkey@0"
I20240501 19:57:06.072894 18697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:06.077500 18697 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:06.078040 18697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:06.078186 18697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:06.078668 18697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:06.079492 18697 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:06.079538 18697 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:06.079586 18697 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:06.079612 18697 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:06.087529 18697 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:35757
I20240501 19:57:06.087563 18824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:35757 every 8 connection(s)
I20240501 19:57:06.088407 18697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 19:57:06.092288 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 18697
I20240501 19:57:06.092459 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 19:57:06.092880 8865 external_mini_cluster.cc:1428] Setting key 96dcca186be3bcc187807d7715e173b0
I20240501 19:57:06.094123 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:06.090015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:55033 (local address 127.8.168.126:40025)
0501 19:57:06.090159 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:06.090164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:06.091052 (+ 888us) server_negotiation.cc:407] Connection header received
0501 19:57:06.091741 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:06.091744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:06.091782 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:06.091857 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:06.093654 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:57:06.093659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:57:06.093662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:57:06.093674 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:57:06.093697 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:57:06.093706 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:57:06.093795 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:06.093930 (+ 135us) server_negotiation.cc:299] Negotiation successful
0501 19:57:06.093976 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20240501 19:57:06.094991 18827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:06.090325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:40025 (local address 127.8.168.65:55033)
0501 19:57:06.090923 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:06.090955 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:06.091574 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:06.092010 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:06.092018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:06.092095 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:06.093392 (+ 1297us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 19:57:06.093394 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:06.093431 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 19:57:06.093492 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:57:06.093788 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:57:06.093796 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:57:06.093802 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 19:57:06.093990 (+ 188us) client_negotiation.cc:240] Negotiation successful
0501 19:57:06.094222 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":96,"threads_started":1}
I20240501 19:57:06.095008 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:40025
--builtin_ntp_servers=127.8.168.84:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20240501 19:57:06.097244 18825 heartbeater.cc:347] Connected to a master server at 127.8.168.126:40025
I20240501 19:57:06.097533 18825 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:06.097975 18825 heartbeater.cc:511] Master 127.8.168.126:40025 requested a full tablet report, sending...
I20240501 19:57:06.099490 18637 ts_manager.cc:194] Registered new tserver with Master: 1aebc4b783634c8fb9d6701869c2b71c (127.8.168.65:35757)
W20240501 19:57:06.198961 18828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:06.199199 18828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:06.199259 18828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:06.202390 18828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:57:06.202451 18828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:06.202535 18828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:57:06.206164 18828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:40025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:06.207062 18828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:06.208154 18828 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:06.216451 18836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:06.216773 18833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:06.216970 18834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:06.217512 18828 server_base.cc:1034] running on GCE node
I20240501 19:57:06.217876 18828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:06.218420 18828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:06.219573 18828 hybrid_clock.cc:648] HybridClock initialized: now 1714593426219557 us; error 31 us; skew 500 ppm
I20240501 19:57:06.221454 18828 webserver.cc:460] Webserver started at http://127.8.168.66:33843/ using document root <none> and password file <none>
I20240501 19:57:06.221971 18828 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:06.222026 18828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:06.222199 18828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:06.223922 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "bc05f3b160894cd4839db093b6aca1a9"
format_stamp: "Formatted at 2024-05-01 19:57:06 on dist-test-slave-tzct"
server_key: "37a5b393a10786bcc83fe8ff6e2290ac"
server_key_iv: "38c7b902e05f20d4769c06b98ca62b82"
server_key_version: "encryptionkey@0"
I20240501 19:57:06.224383 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "bc05f3b160894cd4839db093b6aca1a9"
format_stamp: "Formatted at 2024-05-01 19:57:06 on dist-test-slave-tzct"
server_key: "37a5b393a10786bcc83fe8ff6e2290ac"
server_key_iv: "38c7b902e05f20d4769c06b98ca62b82"
server_key_version: "encryptionkey@0"
I20240501 19:57:06.227533 18828 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 19:57:06.229658 18842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:06.230636 18828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:06.230737 18828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "bc05f3b160894cd4839db093b6aca1a9"
format_stamp: "Formatted at 2024-05-01 19:57:06 on dist-test-slave-tzct"
server_key: "37a5b393a10786bcc83fe8ff6e2290ac"
server_key_iv: "38c7b902e05f20d4769c06b98ca62b82"
server_key_version: "encryptionkey@0"
I20240501 19:57:06.230818 18828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:06.236626 18828 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:06.237349 18828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:06.237501 18828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:06.238000 18828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:06.239044 18828 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:06.239120 18828 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:06.239174 18828 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:06.239200 18828 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:06.247673 18828 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:40543
I20240501 19:57:06.247699 18955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:40543 every 8 connection(s)
I20240501 19:57:06.248513 18828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 19:57:06.250310 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 18828
I20240501 19:57:06.250416 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 19:57:06.250645 8865 external_mini_cluster.cc:1428] Setting key 1d8f99b98b2dac96e215c2d54408ba86
I20240501 19:57:06.253422 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:40025
--builtin_ntp_servers=127.8.168.84:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20240501 19:57:06.255178 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:06.250094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:43171 (local address 127.8.168.126:40025)
0501 19:57:06.250239 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:06.250243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:06.251235 (+ 992us) server_negotiation.cc:407] Connection header received
0501 19:57:06.252267 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:06.252270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:06.252338 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:06.252413 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:06.254650 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:57:06.254654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:57:06.254656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:57:06.254667 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:57:06.254689 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:57:06.254697 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:57:06.254793 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:06.254945 (+ 152us) server_negotiation.cc:299] Negotiation successful
0501 19:57:06.254989 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 19:57:06.255822 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:06.250396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:40025 (local address 127.8.168.66:43171)
0501 19:57:06.251102 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:06.251137 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:06.251817 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:06.253042 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:06.253053 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:06.253156 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:06.254366 (+ 1210us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 19:57:06.254368 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:06.254415 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 19:57:06.254489 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:57:06.254799 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:57:06.254808 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:57:06.254814 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 19:57:06.255000 (+ 186us) client_negotiation.cc:240] Negotiation successful
0501 19:57:06.255216 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":94,"threads_started":1}
I20240501 19:57:06.256773 18956 heartbeater.cc:347] Connected to a master server at 127.8.168.126:40025
I20240501 19:57:06.257014 18956 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:06.257476 18956 heartbeater.cc:511] Master 127.8.168.126:40025 requested a full tablet report, sending...
I20240501 19:57:06.258530 18637 ts_manager.cc:194] Registered new tserver with Master: bc05f3b160894cd4839db093b6aca1a9 (127.8.168.66:40543)
W20240501 19:57:06.358112 18959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:06.358341 18959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:06.358393 18959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:06.361373 18959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:57:06.361428 18959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:06.361500 18959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:57:06.365056 18959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:40025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:06.365906 18959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:06.367034 18959 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:06.374893 18967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:06.375458 18965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:06.376063 18959 server_base.cc:1034] running on GCE node
W20240501 19:57:06.376471 18964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:06.376957 18959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:06.377566 18959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:06.378739 18959 hybrid_clock.cc:648] HybridClock initialized: now 1714593426378714 us; error 38 us; skew 500 ppm
I20240501 19:57:06.380674 18959 webserver.cc:460] Webserver started at http://127.8.168.67:33037/ using document root <none> and password file <none>
I20240501 19:57:06.381255 18959 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:06.381336 18959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:06.381495 18959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:06.383284 18959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "0947c1338cc14fffb5b9b98442b9a9e3"
format_stamp: "Formatted at 2024-05-01 19:57:06 on dist-test-slave-tzct"
server_key: "b0e5610d74dff6825803d7294461a726"
server_key_iv: "bff0c77821005084f64986669b2f65ff"
server_key_version: "encryptionkey@0"
I20240501 19:57:06.383735 18959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "0947c1338cc14fffb5b9b98442b9a9e3"
format_stamp: "Formatted at 2024-05-01 19:57:06 on dist-test-slave-tzct"
server_key: "b0e5610d74dff6825803d7294461a726"
server_key_iv: "bff0c77821005084f64986669b2f65ff"
server_key_version: "encryptionkey@0"
I20240501 19:57:06.387089 18959 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:57:06.389379 18973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:06.390441 18959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:57:06.390568 18959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "0947c1338cc14fffb5b9b98442b9a9e3"
format_stamp: "Formatted at 2024-05-01 19:57:06 on dist-test-slave-tzct"
server_key: "b0e5610d74dff6825803d7294461a726"
server_key_iv: "bff0c77821005084f64986669b2f65ff"
server_key_version: "encryptionkey@0"
I20240501 19:57:06.390659 18959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:06.395476 18959 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:06.396045 18959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:06.396194 18959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:06.396670 18959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:06.397579 18959 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:06.397625 18959 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:06.397675 18959 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:06.397701 18959 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:06.405995 18959 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:33501
I20240501 19:57:06.406019 19086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:33501 every 8 connection(s)
I20240501 19:57:06.406893 18959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 19:57:06.409050 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 18959
I20240501 19:57:06.409176 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 19:57:06.409404 8865 external_mini_cluster.cc:1428] Setting key 9acf4b275ef5dca87229fd036e4b8d0c
I20240501 19:57:06.412499 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:06.408458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:51417 (local address 127.8.168.126:40025)
0501 19:57:06.408575 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:06.408578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:06.409563 (+ 985us) server_negotiation.cc:407] Connection header received
0501 19:57:06.410375 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:06.410379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:06.410422 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:06.410498 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:06.412082 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:57:06.412085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:57:06.412087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 19:57:06.412097 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:57:06.412117 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 19:57:06.412124 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:57:06.412203 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:06.412326 (+ 123us) server_negotiation.cc:299] Negotiation successful
0501 19:57:06.412368 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240501 19:57:06.413213 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:06.408714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:40025 (local address 127.8.168.67:51417)
0501 19:57:06.409413 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:06.409455 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:06.410158 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:06.410657 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:06.410667 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:06.410778 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:06.411850 (+ 1072us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 19:57:06.411852 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:06.411886 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 19:57:06.411943 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:57:06.412195 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:57:06.412202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:57:06.412208 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 19:57:06.412383 (+ 175us) client_negotiation.cc:240] Negotiation successful
0501 19:57:06.412593 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":94,"threads_started":1}
I20240501 19:57:06.414113 19087 heartbeater.cc:347] Connected to a master server at 127.8.168.126:40025
I20240501 19:57:06.414319 19087 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:06.414669 19087 heartbeater.cc:511] Master 127.8.168.126:40025 requested a full tablet report, sending...
I20240501 19:57:06.415561 18637 ts_manager.cc:194] Registered new tserver with Master: 0947c1338cc14fffb5b9b98442b9a9e3 (127.8.168.67:33501)
I20240501 19:57:06.422878 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240501 19:57:06.425417 18676 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:06.424515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.8.168.126:40025)
0501 19:57:06.424634 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:06.424637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:06.424783 (+ 146us) server_negotiation.cc:407] Connection header received
0501 19:57:06.424957 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:06.424960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:06.425040 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:06.425117 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:06.425314 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57322: BlockingRecv error: recv got EOF from 127.0.0.1:57322 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20240501 19:57:06.425535 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 18697
I20240501 19:57:06.429941 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 18828
I20240501 19:57:06.434962 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 18959
I20240501 19:57:06.438997 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 18606
2024-05-01T19:57:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (729 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 19:57:06 dist-test-slave-tzct krb5kdc[19097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 19:57:06 dist-test-slave-tzct krb5kdc[19097](info): set up 2 sockets
May 01 19:57:06 dist-test-slave-tzct krb5kdc[19097](info): commencing operation
krb5kdc: starting...
W20240501 19:57:07.325819 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.868s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 19:57:07 dist-test-slave-tzct krb5kdc[19097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T19:57:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:57:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 19:57:07.437320 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:37217
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:46395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:37217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:57:07.538210 19110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:07.538468 19110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:07.538542 19110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:07.541603 19110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:57:07.541666 19110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:57:07.541697 19110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:07.541721 19110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:57:07.541743 19110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:57:07.545779 19110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:46395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:37217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:37217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:07.546648 19110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:07.547463 19110 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:07.554447 19116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:07.555024 19115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:07.555588 19118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:07.556130 19110 server_base.cc:1034] running on GCE node
I20240501 19:57:07.556684 19110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:07.557624 19110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:07.558774 19110 hybrid_clock.cc:648] HybridClock initialized: now 1714593427558756 us; error 33 us; skew 500 ppm
May 01 19:57:07 dist-test-slave-tzct krb5kdc[19097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593427, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:57:07.561405 19110 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 19:57:07.562309 19110 webserver.cc:460] Webserver started at http://127.8.168.126:42457/ using document root <none> and password file <none>
I20240501 19:57:07.562794 19110 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:07.562841 19110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:07.563001 19110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:07.564749 19110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "f918163ed0324df2885b79ba7c1af7cf"
format_stamp: "Formatted at 2024-05-01 19:57:07 on dist-test-slave-tzct"
server_key: "d524be043cb949aef075cab27317fe8a"
server_key_iv: "45bbc26224f8aa6a3289b6a0ed9bd2d4"
server_key_version: "encryptionkey@0"
I20240501 19:57:07.565199 19110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "f918163ed0324df2885b79ba7c1af7cf"
format_stamp: "Formatted at 2024-05-01 19:57:07 on dist-test-slave-tzct"
server_key: "d524be043cb949aef075cab27317fe8a"
server_key_iv: "45bbc26224f8aa6a3289b6a0ed9bd2d4"
server_key_version: "encryptionkey@0"
I20240501 19:57:07.568300 19110 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240501 19:57:07.570356 19125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:07.571301 19110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:07.571409 19110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "f918163ed0324df2885b79ba7c1af7cf"
format_stamp: "Formatted at 2024-05-01 19:57:07 on dist-test-slave-tzct"
server_key: "d524be043cb949aef075cab27317fe8a"
server_key_iv: "45bbc26224f8aa6a3289b6a0ed9bd2d4"
server_key_version: "encryptionkey@0"
I20240501 19:57:07.571497 19110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:07.597270 19110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:07.599850 19110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:07.599987 19110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:07.606315 19110 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:37217
I20240501 19:57:07.606331 19177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:37217 every 8 connection(s)
I20240501 19:57:07.607551 19110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 19:57:07.609943 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:07.612315 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 19110
I20240501 19:57:07.612416 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 19:57:07.612970 8865 external_mini_cluster.cc:1428] Setting key ff0e942e16936384da5fe098593dd4a0
I20240501 19:57:07.615227 19178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf: Bootstrap starting.
I20240501 19:57:07.617456 19178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:07.618139 19178 log.cc:824] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:07.620091 19178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf: No bootstrap required, opened a new log
May 01 19:57:07 dist-test-slave-tzct krb5kdc[19097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:57:07.623713 19178 raft_consensus.cc:348] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f918163ed0324df2885b79ba7c1af7cf" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 37217 } }
I20240501 19:57:07.623927 19178 raft_consensus.cc:374] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:07.623999 19178 raft_consensus.cc:729] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f918163ed0324df2885b79ba7c1af7cf, State: Initialized, Role: FOLLOWER
I20240501 19:57:07.624468 19178 consensus_queue.cc:260] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f918163ed0324df2885b79ba7c1af7cf" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 37217 } }
I20240501 19:57:07.624632 19178 raft_consensus.cc:388] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:57:07.624696 19178 raft_consensus.cc:482] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:57:07.624819 19178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:07.625412 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:07.614160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34360 (local address 127.8.168.126:37217)
0501 19:57:07.614514 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:07.614523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:07.614564 (+ 41us) server_negotiation.cc:407] Connection header received
0501 19:57:07.615115 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:07.615132 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:07.615415 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:07.615712 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:07.616674 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:07.617473 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:07.618110 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:07.618383 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:07.620803 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:57:07.620822 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:57:07.620835 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:57:07.620876 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:57:07.622820 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:57:07.623214 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:57:07.623220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:57:07.623225 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:57:07.623324 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:57:07.623516 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:57:07.623519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:57:07.623520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:57:07.623944 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:57:07.624169 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:07.624382 (+ 213us) server_negotiation.cc:299] Negotiation successful
0501 19:57:07.624635 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":102,"threads_started":1}
I20240501 19:57:07.625802 19178 raft_consensus.cc:504] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f918163ed0324df2885b79ba7c1af7cf" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 37217 } }
I20240501 19:57:07.626116 19178 leader_election.cc:302] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f918163ed0324df2885b79ba7c1af7cf; no voters:
I20240501 19:57:07.626396 19178 leader_election.cc:288] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:57:07.626464 19183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:07.626678 19183 raft_consensus.cc:686] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [term 1 LEADER]: Becoming Leader. State: Replica: f918163ed0324df2885b79ba7c1af7cf, State: Running, Role: LEADER
I20240501 19:57:07.626976 19183 consensus_queue.cc:237] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f918163ed0324df2885b79ba7c1af7cf" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 37217 } }
I20240501 19:57:07.627454 19178 sys_catalog.cc:564] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:57:07.628451 19185 sys_catalog.cc:455] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f918163ed0324df2885b79ba7c1af7cf. Latest consensus state: current_term: 1 leader_uuid: "f918163ed0324df2885b79ba7c1af7cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f918163ed0324df2885b79ba7c1af7cf" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 37217 } } }
I20240501 19:57:07.628583 19185 sys_catalog.cc:458] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:07.628872 19184 sys_catalog.cc:455] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f918163ed0324df2885b79ba7c1af7cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f918163ed0324df2885b79ba7c1af7cf" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 37217 } } }
I20240501 19:57:07.628963 19184 sys_catalog.cc:458] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:07.628945 19192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:57:07.631385 19192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:57:07.635825 19192 catalog_manager.cc:1347] Generated new cluster ID: bfb8380f12694816b3073139113b56be
I20240501 19:57:07.635890 19192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:57:07.652491 19192 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:57:07.653760 19192 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:57:07.666424 19192 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf: Generated new TSK 0
I20240501 19:57:07.667239 19192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
I20240501 19:57:07.693701 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:37217
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:46395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:57:07.794761 19204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:07.794977 19204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:07.795038 19204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:07.798060 19204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:57:07.798116 19204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:07.798211 19204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 19:57:07.802125 19204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:46395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:37217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:07.802934 19204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:07.803726 19204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:07.811215 19210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:07.811723 19212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:07.812307 19209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:07.812817 19204 server_base.cc:1034] running on GCE node
I20240501 19:57:07.813163 19204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:07.813668 19204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:07.814813 19204 hybrid_clock.cc:648] HybridClock initialized: now 1714593427814799 us; error 28 us; skew 500 ppm
May 01 19:57:07 dist-test-slave-tzct krb5kdc[19097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593427, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:57:07.817222 19204 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 19:57:07.818053 19204 webserver.cc:460] Webserver started at http://127.8.168.65:41565/ using document root <none> and password file <none>
I20240501 19:57:07.818535 19204 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:07.818586 19204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:07.818761 19204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:07.820439 19204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "93f4cb7ff3244b3b9b38eeb8a895d302"
format_stamp: "Formatted at 2024-05-01 19:57:07 on dist-test-slave-tzct"
server_key: "3d5d0e49a5623d1511d1fc89d0b9e312"
server_key_iv: "336f55cf81298376afd136465a6a71f3"
server_key_version: "encryptionkey@0"
I20240501 19:57:07.820915 19204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "93f4cb7ff3244b3b9b38eeb8a895d302"
format_stamp: "Formatted at 2024-05-01 19:57:07 on dist-test-slave-tzct"
server_key: "3d5d0e49a5623d1511d1fc89d0b9e312"
server_key_iv: "336f55cf81298376afd136465a6a71f3"
server_key_version: "encryptionkey@0"
I20240501 19:57:07.823943 19204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240501 19:57:07.825999 19219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:07.826939 19204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:07.827041 19204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "93f4cb7ff3244b3b9b38eeb8a895d302"
format_stamp: "Formatted at 2024-05-01 19:57:07 on dist-test-slave-tzct"
server_key: "3d5d0e49a5623d1511d1fc89d0b9e312"
server_key_iv: "336f55cf81298376afd136465a6a71f3"
server_key_version: "encryptionkey@0"
I20240501 19:57:07.827127 19204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:07.845521 19204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:07.848093 19204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:07.848255 19204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:07.848784 19204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:07.849570 19204 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:07.849617 19204 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:07.849655 19204 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:07.849684 19204 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:07.857717 19204 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:42379
I20240501 19:57:07.857746 19332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:42379 every 8 connection(s)
I20240501 19:57:07.858580 19204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
May 01 19:57:07 dist-test-slave-tzct krb5kdc[19097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593427, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:57:07.868680 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 19204
I20240501 19:57:07.868835 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 19:57:07.869087 8865 external_mini_cluster.cc:1428] Setting key 177724638f48173f3bfbd6a3fa93c938
I20240501 19:57:07.869624 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:07.860166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:58959 (local address 127.8.168.126:37217)
0501 19:57:07.860293 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:07.860296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:07.860928 (+ 632us) server_negotiation.cc:407] Connection header received
0501 19:57:07.862099 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:07.862103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:07.862161 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:07.862235 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:07.863437 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:07.863940 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:07.864554 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:07.864766 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:07.866734 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:57:07.866754 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:57:07.866756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:57:07.866780 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:57:07.867926 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:57:07.868340 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:57:07.868342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:57:07.868343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:57:07.868380 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:57:07.868580 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:57:07.868584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:57:07.868585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:57:07.868795 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:57:07.868893 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:07.869375 (+ 482us) server_negotiation.cc:299] Negotiation successful
0501 19:57:07.869494 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 19:57:07.870220 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:07.860398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:37217 (local address 127.8.168.65:58959)
0501 19:57:07.860805 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:07.860840 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:07.861901 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:07.862395 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:07.862403 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:07.862760 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:07.863298 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:07.863310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:07.864050 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:07.864053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:07.864443 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:07.864450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:07.864646 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:07.865230 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:57:07.865253 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:07.866585 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:57:07.868025 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:57:07.868029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:57:07.868037 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:57:07.868266 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:57:07.868438 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:57:07.868439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:57:07.868441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:57:07.868508 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:57:07.868888 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:57:07.868893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:57:07.869120 (+ 227us) client_negotiation.cc:769] Sending connection context
0501 19:57:07.869312 (+ 192us) client_negotiation.cc:240] Negotiation successful
0501 19:57:07.869549 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":89,"threads_started":1}
I20240501 19:57:07.871260 19333 heartbeater.cc:347] Connected to a master server at 127.8.168.126:37217
I20240501 19:57:07.871451 19333 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:07.871898 19333 heartbeater.cc:511] Master 127.8.168.126:37217 requested a full tablet report, sending...
I20240501 19:57:07.873328 19142 ts_manager.cc:194] Registered new tserver with Master: 93f4cb7ff3244b3b9b38eeb8a895d302 (127.8.168.65:42379)
I20240501 19:57:07.874861 19142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:58959
WARNING: no policy specified for kudu/127.8.168.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
I20240501 19:57:07.893258 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:37217
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:46395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:57:07.995719 19338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:07.995966 19338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:07.996022 19338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:07.999042 19338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:57:07.999107 19338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:07.999184 19338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 19:57:08.003262 19338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:46395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:37217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:08.004083 19338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:08.004886 19338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:08.012748 19344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:08.013306 19343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:08.013424 19346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:08.013988 19338 server_base.cc:1034] running on GCE node
I20240501 19:57:08.014341 19338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:08.014844 19338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:08.015975 19338 hybrid_clock.cc:648] HybridClock initialized: now 1714593428015958 us; error 32 us; skew 500 ppm
May 01 19:57:08 dist-test-slave-tzct krb5kdc[19097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593428, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:57:08.018541 19338 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 19:57:08.019398 19338 webserver.cc:460] Webserver started at http://127.8.168.66:41205/ using document root <none> and password file <none>
I20240501 19:57:08.019903 19338 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:08.019953 19338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:08.020110 19338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:08.021900 19338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "23e060e6536b4717895ef3e72ee68157"
format_stamp: "Formatted at 2024-05-01 19:57:08 on dist-test-slave-tzct"
server_key: "4bd5bad9bc72bd705de7ad44ec2f384f"
server_key_iv: "4a933d8da2f3fb53704ac655bdf8d737"
server_key_version: "encryptionkey@0"
I20240501 19:57:08.022338 19338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "23e060e6536b4717895ef3e72ee68157"
format_stamp: "Formatted at 2024-05-01 19:57:08 on dist-test-slave-tzct"
server_key: "4bd5bad9bc72bd705de7ad44ec2f384f"
server_key_iv: "4a933d8da2f3fb53704ac655bdf8d737"
server_key_version: "encryptionkey@0"
I20240501 19:57:08.025625 19338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:57:08.027750 19353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:08.028831 19338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:57:08.028952 19338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "23e060e6536b4717895ef3e72ee68157"
format_stamp: "Formatted at 2024-05-01 19:57:08 on dist-test-slave-tzct"
server_key: "4bd5bad9bc72bd705de7ad44ec2f384f"
server_key_iv: "4a933d8da2f3fb53704ac655bdf8d737"
server_key_version: "encryptionkey@0"
I20240501 19:57:08.029053 19338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:08.047977 19338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:08.050731 19338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:08.050896 19338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:08.051455 19338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:08.052258 19338 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:08.052305 19338 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:08.052348 19338 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:08.052376 19338 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:08.060670 19338 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:45563
I20240501 19:57:08.060691 19466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:45563 every 8 connection(s)
I20240501 19:57:08.061620 19338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 19:57:08.067891 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 19338
I20240501 19:57:08.068030 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 19:57:08.068295 8865 external_mini_cluster.cc:1428] Setting key 61ff90f39658975a77cd876ec6051265
May 01 19:57:08 dist-test-slave-tzct krb5kdc[19097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593428, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:57:08.073657 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:08.063311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:41729 (local address 127.8.168.126:37217)
0501 19:57:08.063453 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:08.063457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:08.064101 (+ 644us) server_negotiation.cc:407] Connection header received
0501 19:57:08.065279 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:08.065282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:08.065333 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:08.065412 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:08.066635 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.067135 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:08.067757 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.067878 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:08.070439 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:57:08.070455 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:57:08.070457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:57:08.070482 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:57:08.071814 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:57:08.072271 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:57:08.072274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:57:08.072276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:57:08.072320 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:57:08.072563 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:57:08.072567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:57:08.072568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:57:08.072773 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:57:08.072864 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:08.073348 (+ 484us) server_negotiation.cc:299] Negotiation successful
0501 19:57:08.073486 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20240501 19:57:08.074239 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:08.063563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:37217 (local address 127.8.168.66:41729)
0501 19:57:08.063957 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:08.063993 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:08.065080 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:08.065551 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:08.065560 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:08.065937 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:08.066497 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:08.066510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.067236 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:08.067238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:08.067659 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:08.067668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.067893 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:08.068529 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:57:08.068566 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:08.070291 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:57:08.071923 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:57:08.071930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:57:08.071941 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:57:08.072180 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:57:08.072398 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:57:08.072401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:57:08.072402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:57:08.072484 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:57:08.072870 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:57:08.072876 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:57:08.073111 (+ 235us) client_negotiation.cc:769] Sending connection context
0501 19:57:08.073297 (+ 186us) client_negotiation.cc:240] Negotiation successful
0501 19:57:08.073565 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":87,"threads_started":1}
I20240501 19:57:08.075309 19467 heartbeater.cc:347] Connected to a master server at 127.8.168.126:37217
I20240501 19:57:08.075558 19467 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:08.076169 19467 heartbeater.cc:511] Master 127.8.168.126:37217 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.168.67@KRBTEST.COM; defaulting to no policy
I20240501 19:57:08.077247 19142 ts_manager.cc:194] Registered new tserver with Master: 23e060e6536b4717895ef3e72ee68157 (127.8.168.66:45563)
I20240501 19:57:08.077800 19142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:41729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
I20240501 19:57:08.091506 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:37217
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:46395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 19:57:08.192413 19472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:08.192633 19472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:08.192687 19472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:08.195744 19472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 19:57:08.195798 19472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:08.195870 19472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 19:57:08.199679 19472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:46395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:37217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:08.200462 19472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:08.201290 19472 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:08.208946 19477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:08.209367 19480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:08.209914 19478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:08.210381 19472 server_base.cc:1034] running on GCE node
I20240501 19:57:08.210765 19472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:08.211341 19472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:08.212491 19472 hybrid_clock.cc:648] HybridClock initialized: now 1714593428212476 us; error 28 us; skew 500 ppm
May 01 19:57:08 dist-test-slave-tzct krb5kdc[19097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593428, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 19:57:08.215094 19472 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 19:57:08.215941 19472 webserver.cc:460] Webserver started at http://127.8.168.67:36723/ using document root <none> and password file <none>
I20240501 19:57:08.216428 19472 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:08.216475 19472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:08.216625 19472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:08.218396 19472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "8e2544ec986541b1b1a1562179428936"
format_stamp: "Formatted at 2024-05-01 19:57:08 on dist-test-slave-tzct"
server_key: "aa8438160aca2973772e0687dd42ea42"
server_key_iv: "f19839401bfdca85e91f21745273e688"
server_key_version: "encryptionkey@0"
I20240501 19:57:08.218842 19472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "8e2544ec986541b1b1a1562179428936"
format_stamp: "Formatted at 2024-05-01 19:57:08 on dist-test-slave-tzct"
server_key: "aa8438160aca2973772e0687dd42ea42"
server_key_iv: "f19839401bfdca85e91f21745273e688"
server_key_version: "encryptionkey@0"
I20240501 19:57:08.221897 19472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:57:08.223917 19487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:08.224882 19472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:57:08.224969 19472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "8e2544ec986541b1b1a1562179428936"
format_stamp: "Formatted at 2024-05-01 19:57:08 on dist-test-slave-tzct"
server_key: "aa8438160aca2973772e0687dd42ea42"
server_key_iv: "f19839401bfdca85e91f21745273e688"
server_key_version: "encryptionkey@0"
I20240501 19:57:08.225042 19472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:08.239837 19472 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:08.242480 19472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:08.242630 19472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:08.243125 19472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:08.243862 19472 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:08.243908 19472 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:08.243958 19472 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:08.243983 19472 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:08.252328 19472 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:44699
I20240501 19:57:08.252378 19600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:44699 every 8 connection(s)
I20240501 19:57:08.253485 19472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 19:57:08.256242 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 19472
I20240501 19:57:08.256350 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 19:57:08.256577 8865 external_mini_cluster.cc:1428] Setting key 80ae123c20e003595d042cadf768c068
May 01 19:57:08 dist-test-slave-tzct krb5kdc[19097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593428, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 19:57:08.265018 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:08.255490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:47661 (local address 127.8.168.126:37217)
0501 19:57:08.255633 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:08.255637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:08.256278 (+ 641us) server_negotiation.cc:407] Connection header received
0501 19:57:08.257659 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:08.257662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:08.257719 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:08.257787 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:08.258977 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.259460 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:08.260046 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.260225 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:08.262228 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:57:08.262244 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:57:08.262246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:57:08.262271 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:57:08.263359 (+ 1088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:57:08.263775 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:57:08.263778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:57:08.263779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:57:08.263824 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:57:08.264055 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:57:08.264058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:57:08.264060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:57:08.264201 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:57:08.264282 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:08.264745 (+ 463us) server_negotiation.cc:299] Negotiation successful
0501 19:57:08.264859 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20240501 19:57:08.265581 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:08.255738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:37217 (local address 127.8.168.67:47661)
0501 19:57:08.256150 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:08.256183 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:08.257459 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:08.257952 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:08.257960 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:08.258320 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0501 19:57:08.258848 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:08.258860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.259562 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:08.259565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:08.259943 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:08.259950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.260151 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:08.260693 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 19:57:08.260740 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0501 19:57:08.262059 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 19:57:08.263459 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:57:08.263464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:57:08.263474 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:57:08.263687 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:57:08.263899 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 19:57:08.263902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 19:57:08.263903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 19:57:08.263977 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 19:57:08.264295 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 19:57:08.264300 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 19:57:08.264506 (+ 206us) client_negotiation.cc:769] Sending connection context
0501 19:57:08.264673 (+ 167us) client_negotiation.cc:240] Negotiation successful
0501 19:57:08.264957 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":90,"threads_started":1}
I20240501 19:57:08.266527 19601 heartbeater.cc:347] Connected to a master server at 127.8.168.126:37217
I20240501 19:57:08.266737 19601 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:08.267192 19601 heartbeater.cc:511] Master 127.8.168.126:37217 requested a full tablet report, sending...
I20240501 19:57:08.268095 19142 ts_manager.cc:194] Registered new tserver with Master: 8e2544ec986541b1b1a1562179428936 (127.8.168.67:44699)
I20240501 19:57:08.268641 19142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:47661
I20240501 19:57:08.270175 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 19:57:08.277856 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:08.271797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34374 (local address 127.8.168.126:37217)
0501 19:57:08.271932 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:08.271936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:08.272053 (+ 117us) server_negotiation.cc:407] Connection header received
0501 19:57:08.272170 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:08.272174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:08.272225 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:08.272289 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0501 19:57:08.273088 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.273574 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:08.274220 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.274401 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:08.275335 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 19:57:08.275350 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 19:57:08.275352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 19:57:08.275373 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0501 19:57:08.276487 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:57:08.276877 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:57:08.276881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:57:08.276882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:57:08.276924 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 19:57:08.277122 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 19:57:08.277126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 19:57:08.277128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 19:57:08.277290 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 19:57:08.277407 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:08.277605 (+ 198us) server_negotiation.cc:299] Negotiation successful
0501 19:57:08.277710 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20240501 19:57:08.280526 19142 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:57:08.283077 19142 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:57:08.295481 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:08.291280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:42379 (local address 127.0.0.1:38150)
0501 19:57:08.291691 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:08.291703 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:08.291778 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:08.292621 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:08.292624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:08.292637 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:57:08.292903 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:08.292911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.294346 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:08.294349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:08.295180 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:08.295188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.295272 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:08.295285 (+ 13us) client_negotiation.cc:769] Sending connection context
0501 19:57:08.295325 (+ 40us) client_negotiation.cc:240] Negotiation successful
0501 19:57:08.295364 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":108,"threads_started":1}
I20240501 19:57:08.295481 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:08.291639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:44699 (local address 127.0.0.1:56276)
0501 19:57:08.292347 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:08.292359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:08.292428 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:08.292980 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:08.292983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:08.293001 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:57:08.293201 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:08.293207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.294119 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:08.294122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:08.294982 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:08.294992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.295090 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:08.295142 (+ 52us) client_negotiation.cc:769] Sending connection context
0501 19:57:08.295233 (+ 91us) client_negotiation.cc:240] Negotiation successful
0501 19:57:08.295315 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":87,"threads_started":1}
I20240501 19:57:08.296219 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:08.292014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.8.168.67:44699)
0501 19:57:08.292548 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:08.292554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:08.292575 (+ 21us) server_negotiation.cc:407] Connection header received
0501 19:57:08.292654 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:08.292659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:08.292850 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:08.292973 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:57:08.293315 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.294002 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:08.295175 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.295816 (+ 641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:08.295932 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:08.295999 (+ 67us) server_negotiation.cc:299] Negotiation successful
0501 19:57:08.296083 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":101,"threads_started":1}
I20240501 19:57:08.296262 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:08.290945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:45563 (local address 127.0.0.1:60040)
0501 19:57:08.291308 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:08.291351 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:08.291758 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:08.292529 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:08.292534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:08.292550 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:57:08.292903 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:08.292911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.294365 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:08.294368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:08.295187 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:08.295198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.295995 (+ 797us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:08.296011 (+ 16us) client_negotiation.cc:769] Sending connection context
0501 19:57:08.296098 (+ 87us) client_negotiation.cc:240] Negotiation successful
0501 19:57:08.296152 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20240501 19:57:08.296628 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:08.291086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60040 (local address 127.8.168.66:45563)
0501 19:57:08.291478 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:08.291485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:08.291511 (+ 26us) server_negotiation.cc:407] Connection header received
0501 19:57:08.291979 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:08.291986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:08.292199 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:08.292333 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:57:08.293049 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.294250 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:08.295350 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.295856 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:08.295931 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:08.296175 (+ 244us) server_negotiation.cc:299] Negotiation successful
0501 19:57:08.296265 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":111,"threads_started":1}
I20240501 19:57:08.296870 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:08.291684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.8.168.65:42379)
0501 19:57:08.292188 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:08.292194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:08.292217 (+ 23us) server_negotiation.cc:407] Connection header received
0501 19:57:08.292305 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:08.292310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:08.292464 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:08.292641 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:57:08.293035 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.294228 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:08.295570 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.296459 (+ 889us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:08.296572 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:08.296637 (+ 65us) server_negotiation.cc:299] Negotiation successful
0501 19:57:08.296742 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":95,"threads_started":1}
I20240501 19:57:08.298821 19267 tablet_service.cc:1467] Processing CreateTablet for tablet 8c827175c6ac44569fb8e1379679c870 (DEFAULT_TABLE table=test-table [id=900743cf82d3409eb127b393342d3bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:08.298776 19535 tablet_service.cc:1467] Processing CreateTablet for tablet 8c827175c6ac44569fb8e1379679c870 (DEFAULT_TABLE table=test-table [id=900743cf82d3409eb127b393342d3bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:08.298772 19401 tablet_service.cc:1467] Processing CreateTablet for tablet 8c827175c6ac44569fb8e1379679c870 (DEFAULT_TABLE table=test-table [id=900743cf82d3409eb127b393342d3bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:08.299584 19267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c827175c6ac44569fb8e1379679c870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:08.299633 19401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c827175c6ac44569fb8e1379679c870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:08.299633 19535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c827175c6ac44569fb8e1379679c870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:08.305049 19618 tablet_bootstrap.cc:492] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302: Bootstrap starting.
I20240501 19:57:08.306255 19620 tablet_bootstrap.cc:492] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936: Bootstrap starting.
I20240501 19:57:08.306540 19619 tablet_bootstrap.cc:492] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157: Bootstrap starting.
I20240501 19:57:08.307188 19618 tablet_bootstrap.cc:654] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:08.307930 19620 tablet_bootstrap.cc:654] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:08.308004 19618 log.cc:824] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:08.308400 19619 tablet_bootstrap.cc:654] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:08.308578 19620 log.cc:824] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:08.309098 19619 log.cc:824] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:08.309854 19618 tablet_bootstrap.cc:492] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302: No bootstrap required, opened a new log
I20240501 19:57:08.310038 19618 ts_tablet_manager.cc:1367] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:57:08.310150 19620 tablet_bootstrap.cc:492] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936: No bootstrap required, opened a new log
I20240501 19:57:08.310411 19620 ts_tablet_manager.cc:1367] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:57:08.310734 19619 tablet_bootstrap.cc:492] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157: No bootstrap required, opened a new log
I20240501 19:57:08.310882 19619 ts_tablet_manager.cc:1367] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:57:08.313323 19620 raft_consensus.cc:348] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e060e6536b4717895ef3e72ee68157" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45563 } } peers { permanent_uuid: "8e2544ec986541b1b1a1562179428936" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44699 } } peers { permanent_uuid: "93f4cb7ff3244b3b9b38eeb8a895d302" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42379 } }
I20240501 19:57:08.313488 19620 raft_consensus.cc:374] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:08.313537 19620 raft_consensus.cc:729] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e2544ec986541b1b1a1562179428936, State: Initialized, Role: FOLLOWER
I20240501 19:57:08.313441 19618 raft_consensus.cc:348] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e060e6536b4717895ef3e72ee68157" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45563 } } peers { permanent_uuid: "8e2544ec986541b1b1a1562179428936" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44699 } } peers { permanent_uuid: "93f4cb7ff3244b3b9b38eeb8a895d302" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42379 } }
I20240501 19:57:08.313620 19618 raft_consensus.cc:374] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:08.313683 19618 raft_consensus.cc:729] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f4cb7ff3244b3b9b38eeb8a895d302, State: Initialized, Role: FOLLOWER
I20240501 19:57:08.313989 19620 consensus_queue.cc:260] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e060e6536b4717895ef3e72ee68157" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45563 } } peers { permanent_uuid: "8e2544ec986541b1b1a1562179428936" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44699 } } peers { permanent_uuid: "93f4cb7ff3244b3b9b38eeb8a895d302" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42379 } }
I20240501 19:57:08.314086 19618 consensus_queue.cc:260] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e060e6536b4717895ef3e72ee68157" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45563 } } peers { permanent_uuid: "8e2544ec986541b1b1a1562179428936" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44699 } } peers { permanent_uuid: "93f4cb7ff3244b3b9b38eeb8a895d302" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42379 } }
I20240501 19:57:08.314245 19619 raft_consensus.cc:348] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e060e6536b4717895ef3e72ee68157" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45563 } } peers { permanent_uuid: "8e2544ec986541b1b1a1562179428936" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44699 } } peers { permanent_uuid: "93f4cb7ff3244b3b9b38eeb8a895d302" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42379 } }
I20240501 19:57:08.314422 19619 raft_consensus.cc:374] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:08.314487 19619 raft_consensus.cc:729] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23e060e6536b4717895ef3e72ee68157, State: Initialized, Role: FOLLOWER
I20240501 19:57:08.314795 19620 ts_tablet_manager.cc:1398] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 19:57:08.314873 19333 heartbeater.cc:503] Master 127.8.168.126:37217 was elected leader, sending a full tablet report...
I20240501 19:57:08.314898 19618 ts_tablet_manager.cc:1398] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 19:57:08.314898 19619 consensus_queue.cc:260] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e060e6536b4717895ef3e72ee68157" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45563 } } peers { permanent_uuid: "8e2544ec986541b1b1a1562179428936" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44699 } } peers { permanent_uuid: "93f4cb7ff3244b3b9b38eeb8a895d302" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42379 } }
I20240501 19:57:08.315109 19601 heartbeater.cc:503] Master 127.8.168.126:37217 was elected leader, sending a full tablet report...
I20240501 19:57:08.315445 19467 heartbeater.cc:503] Master 127.8.168.126:37217 was elected leader, sending a full tablet report...
I20240501 19:57:08.315634 19619 ts_tablet_manager.cc:1398] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20240501 19:57:08.359614 19334 tablet.cc:2369] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:57:08.504561 19602 tablet.cc:2369] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:57:08.558183 19625 raft_consensus.cc:482] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:08.558457 19625 raft_consensus.cc:504] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e060e6536b4717895ef3e72ee68157" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45563 } } peers { permanent_uuid: "8e2544ec986541b1b1a1562179428936" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44699 } } peers { permanent_uuid: "93f4cb7ff3244b3b9b38eeb8a895d302" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42379 } }
I20240501 19:57:08.560123 19625 leader_election.cc:288] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23e060e6536b4717895ef3e72ee68157 (127.8.168.66:45563), 8e2544ec986541b1b1a1562179428936 (127.8.168.67:44699)
W20240501 19:57:08.562647 19468 tablet.cc:2369] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:57:08.563299 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:08.560398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:45563 (local address 127.8.168.65:47885)
0501 19:57:08.560650 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:08.560665 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:08.560815 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:08.561195 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:08.561197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:08.561224 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:57:08.561472 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:08.561477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.562165 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:08.562168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:08.562882 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:08.562892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.563004 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:08.563026 (+ 22us) client_negotiation.cc:769] Sending connection context
0501 19:57:08.563081 (+ 55us) client_negotiation.cc:240] Negotiation successful
0501 19:57:08.563140 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"spinlock_wait_cycles":39808,"thread_start_us":84,"threads_started":1}
I20240501 19:57:08.563541 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:08.560814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:44699 (local address 127.8.168.65:51731)
0501 19:57:08.561050 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:08.561112 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0501 19:57:08.561249 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:57:08.561491 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 19:57:08.561493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 19:57:08.561508 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 19:57:08.561735 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:08.561741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.562433 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 19:57:08.562436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 19:57:08.563142 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 19:57:08.563152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.563261 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:08.563277 (+ 16us) client_negotiation.cc:769] Sending connection context
0501 19:57:08.563323 (+ 46us) client_negotiation.cc:240] Negotiation successful
0501 19:57:08.563378 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"spinlock_wait_cycles":2304,"thread_start_us":88,"threads_started":1}
I20240501 19:57:08.563724 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:08.560535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:47885 (local address 127.8.168.66:45563)
0501 19:57:08.560677 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:08.560680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:08.560744 (+ 64us) server_negotiation.cc:407] Connection header received
0501 19:57:08.560957 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:08.560961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:08.561015 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:08.561077 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:57:08.561587 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.562062 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:08.563007 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.563456 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:08.563492 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:08.563543 (+ 51us) server_negotiation.cc:299] Negotiation successful
0501 19:57:08.563634 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20240501 19:57:08.563953 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:08.560909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:51731 (local address 127.8.168.67:44699)
0501 19:57:08.561027 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:08.561031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:08.561196 (+ 165us) server_negotiation.cc:407] Connection header received
0501 19:57:08.561350 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:08.561354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:08.561409 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:08.561470 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:57:08.561844 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.562318 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:08.563268 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.563727 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:08.563759 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:08.563816 (+ 57us) server_negotiation.cc:299] Negotiation successful
0501 19:57:08.563866 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20240501 19:57:08.564648 19555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c827175c6ac44569fb8e1379679c870" candidate_uuid: "93f4cb7ff3244b3b9b38eeb8a895d302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e2544ec986541b1b1a1562179428936" is_pre_election: true
I20240501 19:57:08.564648 19421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c827175c6ac44569fb8e1379679c870" candidate_uuid: "93f4cb7ff3244b3b9b38eeb8a895d302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23e060e6536b4717895ef3e72ee68157" is_pre_election: true
I20240501 19:57:08.564986 19421 raft_consensus.cc:2454] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f4cb7ff3244b3b9b38eeb8a895d302 in term 0.
I20240501 19:57:08.564986 19555 raft_consensus.cc:2454] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f4cb7ff3244b3b9b38eeb8a895d302 in term 0.
I20240501 19:57:08.565446 19222 leader_election.cc:302] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23e060e6536b4717895ef3e72ee68157, 93f4cb7ff3244b3b9b38eeb8a895d302; no voters:
I20240501 19:57:08.565686 19625 raft_consensus.cc:2781] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:08.565758 19625 raft_consensus.cc:482] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:08.565819 19625 raft_consensus.cc:3037] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.566735 19625 raft_consensus.cc:504] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e060e6536b4717895ef3e72ee68157" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45563 } } peers { permanent_uuid: "8e2544ec986541b1b1a1562179428936" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44699 } } peers { permanent_uuid: "93f4cb7ff3244b3b9b38eeb8a895d302" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42379 } }
I20240501 19:57:08.567060 19625 leader_election.cc:288] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [CANDIDATE]: Term 1 election: Requested vote from peers 23e060e6536b4717895ef3e72ee68157 (127.8.168.66:45563), 8e2544ec986541b1b1a1562179428936 (127.8.168.67:44699)
I20240501 19:57:08.567476 19421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c827175c6ac44569fb8e1379679c870" candidate_uuid: "93f4cb7ff3244b3b9b38eeb8a895d302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23e060e6536b4717895ef3e72ee68157"
I20240501 19:57:08.567490 19555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c827175c6ac44569fb8e1379679c870" candidate_uuid: "93f4cb7ff3244b3b9b38eeb8a895d302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e2544ec986541b1b1a1562179428936"
I20240501 19:57:08.567628 19421 raft_consensus.cc:3037] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.567628 19555 raft_consensus.cc:3037] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:08.568907 19555 raft_consensus.cc:2454] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f4cb7ff3244b3b9b38eeb8a895d302 in term 1.
I20240501 19:57:08.568907 19421 raft_consensus.cc:2454] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f4cb7ff3244b3b9b38eeb8a895d302 in term 1.
I20240501 19:57:08.569227 19222 leader_election.cc:302] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e2544ec986541b1b1a1562179428936, 93f4cb7ff3244b3b9b38eeb8a895d302; no voters:
I20240501 19:57:08.569397 19625 raft_consensus.cc:2781] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:08.569594 19625 raft_consensus.cc:686] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [term 1 LEADER]: Becoming Leader. State: Replica: 93f4cb7ff3244b3b9b38eeb8a895d302, State: Running, Role: LEADER
I20240501 19:57:08.569923 19625 consensus_queue.cc:237] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e060e6536b4717895ef3e72ee68157" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45563 } } peers { permanent_uuid: "8e2544ec986541b1b1a1562179428936" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44699 } } peers { permanent_uuid: "93f4cb7ff3244b3b9b38eeb8a895d302" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42379 } }
I20240501 19:57:08.572890 19140 catalog_manager.cc:5484] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f4cb7ff3244b3b9b38eeb8a895d302 (127.8.168.65). New cstate: current_term: 1 leader_uuid: "93f4cb7ff3244b3b9b38eeb8a895d302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23e060e6536b4717895ef3e72ee68157" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e2544ec986541b1b1a1562179428936" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f4cb7ff3244b3b9b38eeb8a895d302" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 42379 } health_report { overall_health: HEALTHY } } }
I20240501 19:57:08.585327 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:08.582556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.8.168.65:42379)
0501 19:57:08.582664 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:08.582667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:08.582680 (+ 13us) server_negotiation.cc:407] Connection header received
0501 19:57:08.582816 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:08.582818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:08.582869 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:08.582956 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 19:57:08.583246 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.583721 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 19:57:08.584313 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 19:57:08.584437 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 19:57:08.584515 (+ 78us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 19:57:08.584981 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 19:57:08.585068 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0501 19:57:08.585153 (+ 85us) server_negotiation.cc:299] Negotiation successful
0501 19:57:08.585196 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20240501 19:57:08.590498 19421 raft_consensus.cc:1261] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157 [term 1 FOLLOWER]: Refusing update from remote peer 93f4cb7ff3244b3b9b38eeb8a895d302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:08.590498 19555 raft_consensus.cc:1261] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936 [term 1 FOLLOWER]: Refusing update from remote peer 93f4cb7ff3244b3b9b38eeb8a895d302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:08.591104 19625 consensus_queue.cc:1035] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23e060e6536b4717895ef3e72ee68157" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 45563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:08.591279 19629 consensus_queue.cc:1035] T 8c827175c6ac44569fb8e1379679c870 P 93f4cb7ff3244b3b9b38eeb8a895d302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e2544ec986541b1b1a1562179428936" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 44699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:08.598305 19634 mvcc.cc:205] Tried to move back new op lower bound from 7022974683499065344 to 7022974683424821248. Current Snapshot: MvccSnapshot[applied={T|T < 7022974683499065344}]
I20240501 19:57:08.605636 19140 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34374:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:57:08.605813 19140 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34374:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 19:57:08.608234 19140 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f918163ed0324df2885b79ba7c1af7cf: Sending DeleteTablet for 3 replicas of tablet 8c827175c6ac44569fb8e1379679c870
I20240501 19:57:08.608994 19535 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c827175c6ac44569fb8e1379679c870 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:08 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:56276
I20240501 19:57:08.609006 19401 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c827175c6ac44569fb8e1379679c870 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:08 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:60040
I20240501 19:57:08.609542 19643 tablet_replica.cc:330] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157: stopping tablet replica
I20240501 19:57:08.609828 19643 raft_consensus.cc:2229] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:08.609894 19267 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c827175c6ac44569fb8e1379679c870 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:08 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:38150
I20240501 19:57:08.610153 19643 raft_consensus.cc:2258] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:08.610690 19642 tablet_replica.cc:330] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936: stopping tablet replica
I20240501 19:57:08.610970 19642 raft_consensus.cc:2229] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:08.610991 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 19204
I20240501 19:57:08.611299 19642 raft_consensus.cc:2258] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:08.612632 19642 ts_tablet_manager.cc:1875] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:08.613855 19643 ts_tablet_manager.cc:1875] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:08.615164 19642 ts_tablet_manager.cc:1888] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:08.615262 19642 log.cc:1196] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/8c827175c6ac44569fb8e1379679c870
I20240501 19:57:08.615614 19642 ts_tablet_manager.cc:1909] T 8c827175c6ac44569fb8e1379679c870 P 8e2544ec986541b1b1a1562179428936: Deleting consensus metadata
I20240501 19:57:08.616214 19643 ts_tablet_manager.cc:1888] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:08.616302 19643 log.cc:1196] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/8c827175c6ac44569fb8e1379679c870
I20240501 19:57:08.616636 19643 ts_tablet_manager.cc:1909] T 8c827175c6ac44569fb8e1379679c870 P 23e060e6536b4717895ef3e72ee68157: Deleting consensus metadata
I20240501 19:57:08.616811 19128 catalog_manager.cc:4830] TS 8e2544ec986541b1b1a1562179428936 (127.8.168.67:44699): tablet 8c827175c6ac44569fb8e1379679c870 (table test-table [id=900743cf82d3409eb127b393342d3bb7]) successfully deleted
I20240501 19:57:08.617758 19128 catalog_manager.cc:4830] TS 23e060e6536b4717895ef3e72ee68157 (127.8.168.66:45563): tablet 8c827175c6ac44569fb8e1379679c870 (table test-table [id=900743cf82d3409eb127b393342d3bb7]) successfully deleted
W20240501 19:57:08.620496 19127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.168.65:42379 (error 108)
I20240501 19:57:08.620594 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 19338
I20240501 19:57:08.622674 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 19:57:08.622415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:42379 (local address 127.0.0.1:38170)
0501 19:57:08.622533 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:08.622603 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20240501 19:57:08.622815 19127 catalog_manager.cc:4573] TS 93f4cb7ff3244b3b9b38eeb8a895d302 (127.8.168.65:42379): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c827175c6ac44569fb8e1379679c870: Network error: Client connection negotiation failed: client connection to 127.8.168.65:42379: connect: Connection refused (error 111)
I20240501 19:57:08.625823 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 19472
I20240501 19:57:08.630228 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 19110
2024-05-01T19:57:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2192 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 19:57:08 dist-test-slave-tzct krb5kdc[19648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 19:57:08 dist-test-slave-tzct krb5kdc[19648](info): set up 2 sockets
May 01 19:57:08 dist-test-slave-tzct krb5kdc[19648](info): commencing operation
krb5kdc: starting...
W20240501 19:57:09.521653 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.871s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 19:57:09 dist-test-slave-tzct krb5kdc[19648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T19:57:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:57:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.168.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.168.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.168.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0/krb5kdc/rangeradmin_127.8.168.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.168.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0/krb5kdc/rangeradmin_127.8.168.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.168.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.168.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.168.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0/krb5kdc/rangerlookup_127.8.168.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.168.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0/krb5kdc/rangerlookup_127.8.168.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.168.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.168.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.168.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0/krb5kdc/HTTP_127.8.168.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.168.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0/krb5kdc/HTTP_127.8.168.84@KRBTEST.COM.keytab.
I20240501 19:57:09.675069 8865 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHyS6Hz/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0/minicluster-data/postgres -l logfile start
2024-05-01 19:57:12.125 UTC [19678] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2024-05-01 19:57:12.125 UTC [19678] LOG: listening on IPv4 address "127.8.168.84", port 46027
2024-05-01 19:57:12.129 UTC [19678] LOG: listening on Unix socket "/tmp/.s.PGSQL.46027"
2024-05-01 19:57:12.148 UTC [19681] LOG: database system was shut down at 2024-05-01 19:57:10 UTC
2024-05-01 19:57:12.152 UTC [19678] LOG: database system is ready to accept connections
I20240501 19:57:12.969442 8865 mini_postgres.cc:96] Postgres bound to 46027
2024-05-01 19:57:12.974 UTC [19689] FATAL: database "slave" does not exist
127.8.168.84:46027 - accepting connections
I20240501 19:57:12.975418 8865 mini_ranger.cc:161] Starting Ranger...
I20240501 19:57:12.989199 8865 mini_ranger.cc:84] Created miniranger Postgres user
I20240501 19:57:13.449153 8865 mini_ranger.cc:87] Created ranger Postgres database
I20240501 19:57:13.449258 8865 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0/minicluster-data/ranger-admin
2024-05-01 19:57:13,655 [I] DB FLAVOR :POSTGRES
2024-05-01 19:57:13,656 [I] --------- Verifying Ranger DB connection ---------
2024-05-01 19:57:13,656 [I] Checking connection..
2024-05-01 19:57:13,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2024-05-01 19:57:13,914 [I] Checking connection passed.
2024-05-01 19:57:13,914 [I] --------- Verifying version history table ---------
2024-05-01 19:57:13,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-01 19:57:14,167 [I] Table x_db_version_h does not exist in database ranger
2024-05-01 19:57:14,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-01 19:57:14,424 [I] Table x_db_version_h does not exist in database ranger
2024-05-01 19:57:14,424 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2024-05-01 19:57:14,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2024-05-01 19:57:14.679 UTC [19753] WARNING: there is no transaction in progress
2024-05-01 19:57:14,684 [I] create_dbversion_catalog.sql file imported successfully
2024-05-01 19:57:14,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-01 19:57:14,909 [I] Table x_db_version_h already exists in database 'ranger'
2024-05-01 19:57:14,909 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-01 19:57:14,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2024-05-01 19:57:15,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2024-05-01 19:57:15,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tzct.c.gcp-upstream.internal','N') ;"
2024-05-01 19:57:15,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-01 19:57:15,856 [I] Table x_portal_user does not exist in database ranger
2024-05-01 19:57:15,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-01 19:57:16,099 [I] Table x_policy_ref_group does not exist in database ranger
2024-05-01 19:57:16,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-01 19:57:16,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-01 19:57:16,568 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2024-05-01 19:57:16,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2024-05-01 19:57:17.536 UTC [19861] WARNING: there is no transaction in progress
2024-05-01 19:57:17.553 UTC [19861] WARNING: there is no transaction in progress
2024-05-01 19:57:17.569 UTC [19861] WARNING: there is no transaction in progress
2024-05-01 19:57:17.584 UTC [19861] WARNING: there is no transaction in progress
2024-05-01 19:57:17.602 UTC [19861] WARNING: there is no transaction in progress
2024-05-01 19:57:17.620 UTC [19861] WARNING: there is no transaction in progress
2024-05-01 19:57:17.674 UTC [19861] WARNING: there is no transaction in progress
2024-05-01 19:57:17.681 UTC [19861] WARNING: there is no transaction in progress
2024-05-01 19:57:17.695 UTC [19861] WARNING: there is no transaction in progress
2024-05-01 19:57:17.705 UTC [19861] WARNING: there is no transaction in progress
2024-05-01 19:57:17.716 UTC [19861] WARNING: there is no transaction in progress
2024-05-01 19:57:17.729 UTC [19861] WARNING: there is no transaction in progress
2024-05-01 19:57:17.738 UTC [19861] WARNING: there is no transaction in progress
2024-05-01 19:57:17.745 UTC [19861] WARNING: there is no transaction in progress
2024-05-01 19:57:17.751 UTC [19861] WARNING: there is no transaction in progress
2024-05-01 19:57:17.757 UTC [19861] WARNING: there is no transaction in progress
2024-05-01 19:57:18,375 [I] ranger_core_db_postgres.sql file imported successfully
2024-05-01 19:57:18,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2024-05-01 19:57:18,623 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2024-05-01 19:57:18,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-01 19:57:18,879 [I] Table x_portal_user already exists in database 'ranger'
2024-05-01 19:57:18,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-01 19:57:19,099 [I] Table x_policy_ref_group already exists in database 'ranger'
2024-05-01 19:57:19,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-01 19:57:19,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-01 19:57:19,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tzct.c.gcp-upstream.internal';"
2024-05-01 19:57:19,823 [I] CORE_DB_SCHEMA import status has been updated
2024-05-01 19:57:19,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:46027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2024-05-01 19:57:20,060 [I] DB_PATCHES have already been applied
I20240501 19:57:20.065491 8865 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHyS6Hz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20240501 19:57:20.065551 8865 mini_ranger.cc:193] Using host: 127.8.168.84
I20240501 19:57:20.068336 8865 mini_ranger.cc:227] Ranger admin URL: http://127.8.168.84:45615
May 01, 2024 7:57:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 01, 2024 7:57:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 01, 2024 7:57:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0/minicluster-data/ranger-admin/ews/webapp
May 01, 2024 7:57:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
May 01, 2024 7:57:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0/minicluster-data/ranger-admin/ews/webapp] .....
May 01, 2024 7:57:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0/minicluster-data/ranger-admin/ews/webapp].
May 01, 2024 7:57:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
May 01, 2024 7:57:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
May 01, 2024 7:57:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.TagREST
May 01, 2024 7:57:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
May 01, 2024 7:57:44 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
May 01, 2024 7:57:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
May 01, 2024 7:57:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
May 01, 2024 7:57:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
May 01, 2024 7:57:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
May 01, 2024 7:57:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
May 01, 2024 7:57:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
May 01, 2024 7:57:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
May 01, 2024 7:57:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
May 01, 2024 7:57:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
May 01, 2024 7:57:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
May 01, 2024 7:57:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
May 01, 2024 7:57:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
May 01, 2024 7:57:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
May 01, 2024 7:57:45 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20240501 19:59:20.911839 8865 mini_ranger.cc:160] Time spent starting Ranger: real 127.936s user 0.016s sys 0.143s
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/security-itest.cc:1013: Failure
Failed
Bad status: Timed out: Failed to start the Ranger service: process with PID 19948 is not yet bound to any port at the specified addresses; last attempt running lsof returned 'Not found: could not find pattern of a bound port in lsof output: p19948
f310
n127.0.0.1:33356->127.8.168.84:46027
f311
n127.0.0.1:33362->127.8.168.84:46027
f312
n127.0.0.1:33376->127.8.168.84:46027
f313
n127.0.0.1:33392->127.8.168.84:46027
f314
n127.0.0.1:33406->127.8.168.84:46027'
I20240501 19:59:20.912632 8865 mini_ranger.cc:66] Stopping Ranger...
I20240501 19:59:20.933331 8865 mini_ranger.cc:68] Stopped Ranger
2024-05-01 19:59:20.933 UTC [19678] LOG: received smart shutdown request
2024-05-01 19:59:20.936 UTC [19678] LOG: background worker "logical replication launcher" (PID 19687) exited with exit code 1
2024-05-01 19:59:20.936 UTC [19682] LOG: shutting down
2024-05-01 19:59:21.085 UTC [19678] LOG: database system is shut down
2024-05-01T19:59:21Z chronyd exiting
I20240501 19:59:21.107147 8865 test_util.cc:175] -----------------------------------------------
I20240501 19:59:21.107225 8865 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1714593353892612-8865-0
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration (132458 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 19:59:21 dist-test-slave-tzct krb5kdc[20077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 19:59:21 dist-test-slave-tzct krb5kdc[20077](info): set up 2 sockets
May 01 19:59:21 dist-test-slave-tzct krb5kdc[20077](info): commencing operation
krb5kdc: starting...
W20240501 19:59:21.976305 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.868s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 19:59:22 dist-test-slave-tzct krb5kdc[20077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T19:59:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:59:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.168.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.168.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.168.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/rangeradmin_127.8.168.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.168.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/rangeradmin_127.8.168.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.168.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.168.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.168.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/rangerlookup_127.8.168.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.168.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/rangerlookup_127.8.168.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.168.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.168.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.168.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/HTTP_127.8.168.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.168.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/HTTP_127.8.168.84@KRBTEST.COM.keytab.
I20240501 19:59:22.129628 8865 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHyS6Hz/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/postgres -l logfile start
2024-05-01 19:59:24.540 UTC [20107] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2024-05-01 19:59:24.540 UTC [20107] LOG: listening on IPv4 address "127.8.168.84", port 52933
2024-05-01 19:59:24.544 UTC [20107] LOG: listening on Unix socket "/tmp/.s.PGSQL.52933"
2024-05-01 19:59:24.564 UTC [20110] LOG: database system was shut down at 2024-05-01 19:59:23 UTC
2024-05-01 19:59:24.567 UTC [20107] LOG: database system is ready to accept connections
I20240501 19:59:25.372388 8865 mini_postgres.cc:96] Postgres bound to 52933
2024-05-01 19:59:25.377 UTC [20118] FATAL: database "slave" does not exist
127.8.168.84:52933 - accepting connections
I20240501 19:59:25.378185 8865 mini_ranger.cc:161] Starting Ranger...
I20240501 19:59:25.391750 8865 mini_ranger.cc:84] Created miniranger Postgres user
I20240501 19:59:25.820852 8865 mini_ranger.cc:87] Created ranger Postgres database
I20240501 19:59:25.820926 8865 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-admin
2024-05-01 19:59:26,031 [I] DB FLAVOR :POSTGRES
2024-05-01 19:59:26,031 [I] --------- Verifying Ranger DB connection ---------
2024-05-01 19:59:26,031 [I] Checking connection..
2024-05-01 19:59:26,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2024-05-01 19:59:26,262 [I] Checking connection passed.
2024-05-01 19:59:26,262 [I] --------- Verifying version history table ---------
2024-05-01 19:59:26,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-01 19:59:26,507 [I] Table x_db_version_h does not exist in database ranger
2024-05-01 19:59:26,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-01 19:59:26,744 [I] Table x_db_version_h does not exist in database ranger
2024-05-01 19:59:26,744 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2024-05-01 19:59:26,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2024-05-01 19:59:26.987 UTC [20182] WARNING: there is no transaction in progress
2024-05-01 19:59:26,993 [I] create_dbversion_catalog.sql file imported successfully
2024-05-01 19:59:26,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-01 19:59:27,231 [I] Table x_db_version_h already exists in database 'ranger'
2024-05-01 19:59:27,231 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-01 19:59:27,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2024-05-01 19:59:27,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2024-05-01 19:59:27,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tzct.c.gcp-upstream.internal','N') ;"
2024-05-01 19:59:27,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-01 19:59:28,171 [I] Table x_portal_user does not exist in database ranger
2024-05-01 19:59:28,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-01 19:59:28,393 [I] Table x_policy_ref_group does not exist in database ranger
2024-05-01 19:59:28,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-01 19:59:28,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-01 19:59:28,881 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2024-05-01 19:59:28,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2024-05-01 19:59:29.820 UTC [20293] WARNING: there is no transaction in progress
2024-05-01 19:59:29.835 UTC [20293] WARNING: there is no transaction in progress
2024-05-01 19:59:29.851 UTC [20293] WARNING: there is no transaction in progress
2024-05-01 19:59:29.866 UTC [20293] WARNING: there is no transaction in progress
2024-05-01 19:59:29.882 UTC [20293] WARNING: there is no transaction in progress
2024-05-01 19:59:29.900 UTC [20293] WARNING: there is no transaction in progress
2024-05-01 19:59:29.957 UTC [20293] WARNING: there is no transaction in progress
2024-05-01 19:59:29.964 UTC [20293] WARNING: there is no transaction in progress
2024-05-01 19:59:29.979 UTC [20293] WARNING: there is no transaction in progress
2024-05-01 19:59:29.989 UTC [20293] WARNING: there is no transaction in progress
2024-05-01 19:59:29.999 UTC [20293] WARNING: there is no transaction in progress
2024-05-01 19:59:30.012 UTC [20293] WARNING: there is no transaction in progress
2024-05-01 19:59:30.022 UTC [20293] WARNING: there is no transaction in progress
2024-05-01 19:59:30.028 UTC [20293] WARNING: there is no transaction in progress
2024-05-01 19:59:30.035 UTC [20293] WARNING: there is no transaction in progress
2024-05-01 19:59:30.042 UTC [20293] WARNING: there is no transaction in progress
2024-05-01 19:59:30,693 [I] ranger_core_db_postgres.sql file imported successfully
2024-05-01 19:59:30,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2024-05-01 19:59:30,922 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2024-05-01 19:59:30,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-01 19:59:31,173 [I] Table x_portal_user already exists in database 'ranger'
2024-05-01 19:59:31,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-01 19:59:31,434 [I] Table x_policy_ref_group already exists in database 'ranger'
2024-05-01 19:59:31,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-01 19:59:31,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-01 19:59:31,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tzct.c.gcp-upstream.internal';"
2024-05-01 19:59:32,132 [I] CORE_DB_SCHEMA import status has been updated
2024-05-01 19:59:32,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2024-05-01 19:59:32,374 [I] DB_PATCHES have already been applied
I20240501 19:59:32.379478 8865 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHyS6Hz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20240501 19:59:32.379551 8865 mini_ranger.cc:193] Using host: 127.8.168.84
I20240501 19:59:32.382278 8865 mini_ranger.cc:227] Ranger admin URL: http://127.8.168.84:44203
May 01, 2024 7:59:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 01, 2024 7:59:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 01, 2024 7:59:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-admin/ews/webapp
May 01, 2024 7:59:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
May 01, 2024 7:59:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-admin/ews/webapp] .....
May 01, 2024 7:59:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-admin/ews/webapp].
May 01, 2024 7:59:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20240501 19:59:33.302173 8865 mini_ranger.cc:160] Time spent starting Ranger: real 7.924s user 0.000s sys 0.009s
May 01, 2024 7:59:53 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
May 01, 2024 7:59:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
May 01, 2024 7:59:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
May 01, 2024 7:59:53 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
May 01, 2024 7:59:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
May 01, 2024 7:59:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
May 01, 2024 7:59:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
May 01, 2024 7:59:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
May 01, 2024 7:59:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
May 01, 2024 7:59:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
May 01, 2024 7:59:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
May 01, 2024 7:59:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
May 01, 2024 7:59:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
May 01, 2024 7:59:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
May 01, 2024 7:59:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
May 01, 2024 7:59:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
May 01, 2024 7:59:53 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
May 01, 2024 7:59:54 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20240501 19:59:54.985745 8865 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.168.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.168.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.168.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/rangerkms_127.8.168.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.168.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/rangerkms_127.8.168.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.168.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.168.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.168.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/HTTP_127.8.168.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.168.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/HTTP_127.8.168.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
May 01 19:59:55 dist-test-slave-tzct krb5kdc[20077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593595, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20240501 19:59:55.081097 8865 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20240501 19:59:55.090438 8865 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20240501 19:59:55.934857 8865 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20240501 19:59:55.934940 8865 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-kms
I20240501 19:59:55.934978 8865 mini_ranger_kms.cc:227] Using postgres at 127.8.168.84:52933
2024-05-01 19:59:56,089 [I] DB FLAVOR :POSTGRES
2024-05-01 19:59:56,089 [I] --------- Verifying Ranger DB connection ---------
2024-05-01 19:59:56,089 [I] Checking connection
2024-05-01 19:59:56,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2024-05-01 19:59:56,336 [I] connection success
2024-05-01 19:59:56,336 [I] --------- Verifying Ranger DB tables ---------
2024-05-01 19:59:56,336 [I] Verifying table ranger_masterkey in database rangerkms
2024-05-01 19:59:56,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2024-05-01 19:59:56,577 [I] Table ranger_masterkey does not exist in database rangerkms
2024-05-01 19:59:56,577 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-01 19:59:56,578 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2024-05-01 19:59:56,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.168.84:52933/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2024-05-01 19:59:56,849 [I] kms_core_db_postgres.sql DB schema imported successfully
I20240501 19:59:57.285370 8865 mini_ranger_kms.cc:310] Created kms service
I20240501 19:59:57.496662 8865 mini_ranger_kms.cc:326] Created kudu user
I20240501 19:59:57.542968 8865 mini_ranger_kms.cc:343] Created rangerkms user
May 01 19:59:57 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593595, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.168.84@KRBTEST.COM
I20240501 19:59:57.920344 8865 mini_ranger_kms.cc:384] Added ranger policy
I20240501 19:59:57.920506 8865 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHyS6Hz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHyS6Hz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20240501 19:59:57.920529 8865 mini_ranger_kms.cc:240] Using host: 127.8.168.84
I20240501 19:59:57.924140 8865 mini_ranger_kms.cc:276] Ranger KMS PID: 20474
I20240501 19:59:57.924242 8865 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.168.84:49677
May 01, 2024 7:59:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 01, 2024 7:59:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 01, 2024 7:59:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
May 01, 2024 7:59:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-kms/ews/webapp] .....
May 01, 2024 7:59:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-kms/ews/webapp].
May 01, 2024 7:59:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20240501 19:59:59.791219 8865 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.710s user 0.007s sys 0.133s
I20240501 19:59:59.791334 8865 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20240501 19:59:59.791397 8865 mini_ranger_kms.cc:401] 127.8.168.84:49677/kms/v1/keys
May 01 19:59:59 dist-test-slave-tzct krb5kdc[20077](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714593599, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.168.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
May 01 20:00:00 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593599, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.168.84@KRBTEST.COM for HTTP/127.8.168.84@KRBTEST.COM
May 01 20:00:00 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593599, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.168.84@KRBTEST.COM for HTTP/127.8.168.84@KRBTEST.COM
May 01 20:00:03 dist-test-slave-tzct krb5kdc[20077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 20:00:03.175522 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:41213
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:41213
--ranger_config_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.168.84:49677/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:03.284631 20525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:03.284915 20525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:03.284986 20525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:03.288026 20525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:00:03.288097 20525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:03.288126 20525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:03.288149 20525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:00:03.288192 20525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:00:03.292544 20525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.168.84:49677/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:41213
--ranger_config_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:41213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:03.293471 20525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:03.294368 20525 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:00:03.302572 20525 server_base.cc:1034] running on GCE node
W20240501 20:00:03.303087 20531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:03.303390 20533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:03.303673 20530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:03.304190 20525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:03.305168 20525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:03.306325 20525 hybrid_clock.cc:648] HybridClock initialized: now 1714593603306307 us; error 34 us; skew 500 ppm
May 01 20:00:03 dist-test-slave-tzct krb5kdc[20077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593603, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:03.308971 20525 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:00:03.309865 20525 webserver.cc:460] Webserver started at http://127.8.168.126:37555/ using document root <none> and password file <none>
I20240501 20:00:03.310372 20525 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:03.310422 20525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:03.310588 20525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 01 20:00:03 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593603, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for HTTP/127.8.168.84@KRBTEST.COM
I20240501 20:00:03.369035 20525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "914b896f08a645f8ad0fa00ac139f6a8"
format_stamp: "Formatted at 2024-05-01 20:00:03 on dist-test-slave-tzct"
server_key: "0a8a3fe2a5733b9557aaef30dda1ecf8"
server_key_iv: "2551c9517cd5c582033da51748845734"
server_key_version: "kuduclusterkey@0"
I20240501 20:00:03.369549 20525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "914b896f08a645f8ad0fa00ac139f6a8"
format_stamp: "Formatted at 2024-05-01 20:00:03 on dist-test-slave-tzct"
server_key: "0a8a3fe2a5733b9557aaef30dda1ecf8"
server_key_iv: "2551c9517cd5c582033da51748845734"
server_key_version: "kuduclusterkey@0"
I20240501 20:00:03.372960 20525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:00:03.390637 20541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:03.392026 20525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:00:03.392164 20525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "914b896f08a645f8ad0fa00ac139f6a8"
format_stamp: "Formatted at 2024-05-01 20:00:03 on dist-test-slave-tzct"
server_key: "0a8a3fe2a5733b9557aaef30dda1ecf8"
server_key_iv: "2551c9517cd5c582033da51748845734"
server_key_version: "kuduclusterkey@0"
I20240501 20:00:03.392313 20525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:03.422518 20525 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:03.426313 20525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:03.426496 20525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:03.435211 20593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:41213 every 8 connection(s)
I20240501 20:00:03.435220 20525 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:41213
I20240501 20:00:03.436436 20525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:00:03.439021 20594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:03.443233 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 20525
I20240501 20:00:03.443361 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:00:03.443614 8865 external_mini_cluster.cc:1428] Setting key 20a015c88f5911bf7d80c51af78bc6d2
I20240501 20:00:03.444077 20594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8: Bootstrap starting.
I20240501 20:00:03.446557 20594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:03.447386 20594 log.cc:824] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:03.449402 20594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8: No bootstrap required, opened a new log
May 01 20:00:03 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:03.452786 20594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914b896f08a645f8ad0fa00ac139f6a8" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41213 } }
I20240501 20:00:03.452982 20594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:03.453049 20594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914b896f08a645f8ad0fa00ac139f6a8, State: Initialized, Role: FOLLOWER
I20240501 20:00:03.453474 20594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914b896f08a645f8ad0fa00ac139f6a8" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41213 } }
I20240501 20:00:03.453598 20594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:00:03.453656 20594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:00:03.453760 20594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:03.454551 20594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914b896f08a645f8ad0fa00ac139f6a8" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41213 } }
I20240501 20:00:03.454861 20594 leader_election.cc:302] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 914b896f08a645f8ad0fa00ac139f6a8; no voters:
I20240501 20:00:03.455140 20594 leader_election.cc:288] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:00:03.455235 20599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:03.455407 20599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [term 1 LEADER]: Becoming Leader. State: Replica: 914b896f08a645f8ad0fa00ac139f6a8, State: Running, Role: LEADER
I20240501 20:00:03.455741 20599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914b896f08a645f8ad0fa00ac139f6a8" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41213 } }
I20240501 20:00:03.456319 20594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:00:03.456487 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:03.445028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57666 (local address 127.8.168.126:41213)
0501 20:00:03.445497 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:03.445508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:03.445550 (+ 42us) server_negotiation.cc:407] Connection header received
0501 20:00:03.446334 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:03.446360 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:03.446738 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:03.447155 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:03.448009 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:03.448851 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:03.449656 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:03.449849 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:03.452203 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:03.452224 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:03.452235 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:03.452268 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:03.454027 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:03.454404 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:03.454409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:03.454413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:03.454474 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:03.454673 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:03.454676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:03.454678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:03.455033 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:03.455264 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:03.455542 (+ 278us) server_negotiation.cc:299] Negotiation successful
0501 20:00:03.455908 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":186,"threads_started":1}
I20240501 20:00:03.458148 20600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "914b896f08a645f8ad0fa00ac139f6a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914b896f08a645f8ad0fa00ac139f6a8" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41213 } } }
I20240501 20:00:03.458252 20600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:03.458545 20594 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20240501 20:00:03.458525 20601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 914b896f08a645f8ad0fa00ac139f6a8. Latest consensus state: current_term: 1 leader_uuid: "914b896f08a645f8ad0fa00ac139f6a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914b896f08a645f8ad0fa00ac139f6a8" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41213 } } }
I20240501 20:00:03.458592 20601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:03.459385 20613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:00:03.465018 20613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:00:03.471259 20613 catalog_manager.cc:1347] Generated new cluster ID: 1ea0985aed5a40faaf07f35ca63dd745
I20240501 20:00:03.471380 20613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:00:03.489418 20613 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:00:03.490427 20613 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:00:03.498670 20613 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8: Generated new TSK 0
I20240501 20:00:03.499541 20613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
I20240501 20:00:04.132268 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.168.84:49677/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:41213
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:04.267555 20639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:04.267805 20639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:04.267882 20639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:04.271121 20639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:04.271198 20639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:04.271301 20639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 20:00:04.275552 20639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.168.84:49677/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:41213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:04.276559 20639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:04.277489 20639 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:04.285660 20644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:04.286309 20639 server_base.cc:1034] running on GCE node
W20240501 20:00:04.286821 20645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:04.287307 20647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:04.287761 20639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:04.288504 20639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:04.289690 20639 hybrid_clock.cc:648] HybridClock initialized: now 1714593604289675 us; error 27 us; skew 500 ppm
May 01 20:00:04 dist-test-slave-tzct krb5kdc[20077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593604, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:04.293056 20639 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 20:00:04.294272 20639 webserver.cc:460] Webserver started at http://127.8.168.65:36535/ using document root <none> and password file <none>
I20240501 20:00:04.294962 20639 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:04.295040 20639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:04.295253 20639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 01 20:00:04 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593604, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for HTTP/127.8.168.84@KRBTEST.COM
I20240501 20:00:04.308909 20639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "289bb9c0d3884c649a1814ed760500bf"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-tzct"
server_key: "52d40152479e9971be11ec1d2e442811"
server_key_iv: "257912a6703ba1ec51d5f70020923752"
server_key_version: "kuduclusterkey@0"
I20240501 20:00:04.309454 20639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "289bb9c0d3884c649a1814ed760500bf"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-tzct"
server_key: "52d40152479e9971be11ec1d2e442811"
server_key_iv: "257912a6703ba1ec51d5f70020923752"
server_key_version: "kuduclusterkey@0"
I20240501 20:00:04.319578 20639 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.003s sys 0.000s
I20240501 20:00:04.335353 20656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:04.336872 20639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:00:04.337009 20639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "289bb9c0d3884c649a1814ed760500bf"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-tzct"
server_key: "52d40152479e9971be11ec1d2e442811"
server_key_iv: "257912a6703ba1ec51d5f70020923752"
server_key_version: "kuduclusterkey@0"
I20240501 20:00:04.337141 20639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:04.356966 20639 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:04.360982 20639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:04.361240 20639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:04.362213 20639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:04.363344 20639 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:04.363406 20639 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:04.363459 20639 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:04.363514 20639 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:04.374850 20639 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:34291
I20240501 20:00:04.376089 20770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:34291 every 8 connection(s)
I20240501 20:00:04.376190 20639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 20:00:04.381996 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 20639
I20240501 20:00:04.382135 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 20:00:04.382376 8865 external_mini_cluster.cc:1428] Setting key 78fe2b786db4b35b943bc637046e023b
May 01 20:00:04 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593604, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:04.394483 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:04.379007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41213 (local address 127.8.168.65:40389)
0501 20:00:04.379448 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:04.379488 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:04.380294 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:04.380866 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:04.380874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:04.381152 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:04.381785 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:04.381804 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:04.382869 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:04.382873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:04.383463 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:04.383474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:04.384074 (+ 600us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:04.384713 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:04.384793 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:04.389118 (+ 4325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:04.391499 (+ 2381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:04.391505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:04.391519 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:04.391861 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:04.392132 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:04.392135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:04.392137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:04.392267 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:04.392758 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:04.392764 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:04.393085 (+ 321us) client_negotiation.cc:769] Sending connection context
0501 20:00:04.393320 (+ 235us) client_negotiation.cc:240] Negotiation successful
0501 20:00:04.393601 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.8.168.66@KRBTEST.COM; defaulting to no policy
I20240501 20:00:04.395205 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:04.378644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:40389 (local address 127.8.168.126:41213)
0501 20:00:04.378935 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:04.378940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:04.379585 (+ 645us) server_negotiation.cc:407] Connection header received
0501 20:00:04.380502 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:04.380506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:04.380568 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:04.380657 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:04.381961 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:04.382713 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:04.383604 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:04.383805 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:04.389362 (+ 5557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:04.389386 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:04.389389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:04.389419 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:04.391340 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:04.391981 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:04.391985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:04.391987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:04.392041 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:04.392379 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:04.392382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:04.392384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:04.392581 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:04.392669 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:04.394900 (+ 2231us) server_negotiation.cc:299] Negotiation successful
0501 20:00:04.395037 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":96,"threads_started":1}
I20240501 20:00:04.396848 20771 heartbeater.cc:347] Connected to a master server at 127.8.168.126:41213
I20240501 20:00:04.397096 20771 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:04.397729 20771 heartbeater.cc:511] Master 127.8.168.126:41213 requested a full tablet report, sending...
I20240501 20:00:04.399533 20558 ts_manager.cc:194] Registered new tserver with Master: 289bb9c0d3884c649a1814ed760500bf (127.8.168.65:34291)
I20240501 20:00:04.401247 20558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:40389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
I20240501 20:00:04.420006 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.168.84:49677/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:41213
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 20:00:04 dist-test-slave-tzct krb5kdc[20077](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714593604, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20240501 20:00:04.589546 20779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:04.589865 20779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:04.589952 20779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:04.594319 20779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:04.594406 20779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:04.594519 20779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 20:00:04.600382 20779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.168.84:49677/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:41213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:04.601637 20779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:04.602808 20779 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:04.613957 20785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:04.614430 20788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:04.615027 20786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:04.615536 20779 server_base.cc:1034] running on GCE node
I20240501 20:00:04.615943 20779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:04.616546 20779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:04.617692 20779 hybrid_clock.cc:648] HybridClock initialized: now 1714593604617671 us; error 65 us; skew 500 ppm
May 01 20:00:04 dist-test-slave-tzct krb5kdc[20077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593604, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:04.620437 20779 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 20:00:04.621418 20779 webserver.cc:460] Webserver started at http://127.8.168.66:46881/ using document root <none> and password file <none>
I20240501 20:00:04.621984 20779 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:04.622040 20779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:04.622208 20779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 01 20:00:04 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593604, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for HTTP/127.8.168.84@KRBTEST.COM
I20240501 20:00:04.642680 20779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "0addd926561444b5943eb0293fbd8eda"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-tzct"
server_key: "44d57f028ae390155e9c9ab241440705"
server_key_iv: "b8f6138babe34d964cfb763a525c46e4"
server_key_version: "kuduclusterkey@0"
I20240501 20:00:04.643369 20779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "0addd926561444b5943eb0293fbd8eda"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-tzct"
server_key: "44d57f028ae390155e9c9ab241440705"
server_key_iv: "b8f6138babe34d964cfb763a525c46e4"
server_key_version: "kuduclusterkey@0"
I20240501 20:00:04.649091 20779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 20:00:04.679534 20795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:04.684615 20779 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.002s sys 0.000s
I20240501 20:00:04.684908 20779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "0addd926561444b5943eb0293fbd8eda"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-tzct"
server_key: "44d57f028ae390155e9c9ab241440705"
server_key_iv: "b8f6138babe34d964cfb763a525c46e4"
server_key_version: "kuduclusterkey@0"
I20240501 20:00:04.685076 20779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:04.714263 20779 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:04.718611 20779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:04.718897 20779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:04.719605 20779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:04.728176 20779 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:04.728509 20779 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20240501 20:00:04.728688 20779 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:04.732007 20779 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20240501 20:00:04.744741 20779 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:35427
I20240501 20:00:04.745776 20779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 20:00:04.746095 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 20779
I20240501 20:00:04.746205 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 20:00:04.746438 8865 external_mini_cluster.cc:1428] Setting key 6eff5528a0c9ba3f74b6b0986b6e2d2f
I20240501 20:00:04.744756 20908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:35427 every 8 connection(s)
WARNING: no policy specified for kudu/127.8.168.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.67@KRBTEST.COM" created.
May 01 20:00:04 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593604, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:04.765977 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:04.749020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41213 (local address 127.8.168.66:40933)
0501 20:00:04.750033 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:04.750069 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:04.750754 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:04.752933 (+ 2179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:04.752946 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:04.753319 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:04.754171 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:04.754193 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:04.755359 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:04.755363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:04.755939 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:04.755951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:04.756612 (+ 661us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:04.757313 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:04.757352 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:04.760638 (+ 3286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:04.762866 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:04.762873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:04.762886 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:04.763210 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:04.763471 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:04.763474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:04.763476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:04.763612 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:04.764096 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:04.764103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:04.764458 (+ 355us) client_negotiation.cc:769] Sending connection context
0501 20:00:04.764700 (+ 242us) client_negotiation.cc:240] Negotiation successful
0501 20:00:04.765055 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":126,"threads_started":1}
I20240501 20:00:04.767045 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:04.748584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:40933 (local address 127.8.168.126:41213)
0501 20:00:04.748701 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:04.748704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:04.750166 (+ 1462us) server_negotiation.cc:407] Connection header received
0501 20:00:04.750957 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:04.750960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:04.751013 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:04.751084 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:04.754384 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:04.755183 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:04.756077 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:04.756317 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:04.760886 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:04.760913 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:04.760916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:04.760946 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:04.762691 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:04.763326 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:04.763330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:04.763332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:04.763387 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:04.763725 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:04.763729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:04.763730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:04.763932 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:04.764036 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:04.766714 (+ 2678us) server_negotiation.cc:299] Negotiation successful
0501 20:00:04.766870 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240501 20:00:04.768215 20909 heartbeater.cc:347] Connected to a master server at 127.8.168.126:41213
I20240501 20:00:04.768522 20909 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:04.769294 20909 heartbeater.cc:511] Master 127.8.168.126:41213 requested a full tablet report, sending...
I20240501 20:00:04.770501 20558 ts_manager.cc:194] Registered new tserver with Master: 0addd926561444b5943eb0293fbd8eda (127.8.168.66:35427)
I20240501 20:00:04.771224 20558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:40933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
I20240501 20:00:04.789325 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.168.84:49677/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:41213
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:04.946208 20914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:04.946528 20914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:04.946604 20914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:04.951135 20914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:04.951231 20914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:04.951349 20914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 20:00:04.957340 20914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.168.84:49677/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:41213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:04.958645 20914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:04.959758 20914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:00:04.971311 20914 server_base.cc:1034] running on GCE node
W20240501 20:00:04.973829 20922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:04.974288 20919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:04.977727 20920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:04.978327 20914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:04.979034 20914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:04.980197 20914 hybrid_clock.cc:648] HybridClock initialized: now 1714593604980165 us; error 52 us; skew 500 ppm
May 01 20:00:04 dist-test-slave-tzct krb5kdc[20077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593604, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:04.983752 20914 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 20:00:04.984968 20914 webserver.cc:460] Webserver started at http://127.8.168.67:41659/ using document root <none> and password file <none>
I20240501 20:00:04.985652 20914 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:04.985767 20914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:04.986080 20914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 01 20:00:04 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593604, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for HTTP/127.8.168.84@KRBTEST.COM
May 01 20:00:04 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593604, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for HTTP/127.8.168.84@KRBTEST.COM
I20240501 20:00:05.005179 20914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "587e967d14484130823c3cba9f68e5d5"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-tzct"
server_key: "921cc37201878d0de338672381b9019a"
server_key_iv: "6a356041447ee44daaf7786348ba0ea4"
server_key_version: "kuduclusterkey@0"
I20240501 20:00:05.006909 20914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "587e967d14484130823c3cba9f68e5d5"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-tzct"
server_key: "921cc37201878d0de338672381b9019a"
server_key_iv: "6a356041447ee44daaf7786348ba0ea4"
server_key_version: "kuduclusterkey@0"
I20240501 20:00:05.011919 20914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20240501 20:00:05.063529 20929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:05.065289 20914 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20240501 20:00:05.065431 20914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "587e967d14484130823c3cba9f68e5d5"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-tzct"
server_key: "921cc37201878d0de338672381b9019a"
server_key_iv: "6a356041447ee44daaf7786348ba0ea4"
server_key_version: "kuduclusterkey@0"
I20240501 20:00:05.065547 20914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:05.100519 20914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:05.115278 20914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:05.115458 20914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:05.115988 20914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:05.116896 20914 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:05.116951 20914 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:05.117002 20914 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:05.117020 20914 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:05.126638 20914 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:46227
I20240501 20:00:05.127924 20914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 20:00:05.132812 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 20914
I20240501 20:00:05.132952 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 20:00:05.133095 21042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:46227 every 8 connection(s)
I20240501 20:00:05.133208 8865 external_mini_cluster.cc:1428] Setting key b836e9582bada727c9124d09ab932bb0
May 01 20:00:05 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593604, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:05.149282 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:05.135969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:42833 (local address 127.8.168.126:41213)
0501 20:00:05.136081 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:05.136086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:05.136100 (+ 14us) server_negotiation.cc:407] Connection header received
0501 20:00:05.136872 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:05.136875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:05.136935 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:05.136982 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:05.138741 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:05.139523 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:05.140404 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:05.140589 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:05.143625 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:05.143649 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:05.143652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:05.143683 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:05.145426 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:05.147590 (+ 2164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:05.147594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:05.147596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:05.147650 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:05.147977 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:05.147980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:05.147982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:05.148168 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:05.148234 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:05.148964 (+ 730us) server_negotiation.cc:299] Negotiation successful
0501 20:00:05.149116 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240501 20:00:05.150879 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:05.134943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41213 (local address 127.8.168.67:42833)
0501 20:00:05.135667 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:05.135716 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:05.136589 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:05.137386 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:05.137398 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:05.137758 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:05.138573 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:05.138591 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:05.139672 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:05.139675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:05.140280 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:05.140288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:05.140863 (+ 575us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:05.141576 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:05.141615 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:05.143429 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:05.145983 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:05.145989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:05.146003 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:05.147475 (+ 1472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:05.147748 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:05.147751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:05.147753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:05.147883 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:05.148308 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:05.148314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:05.148636 (+ 322us) client_negotiation.cc:769] Sending connection context
0501 20:00:05.149743 (+ 1107us) client_negotiation.cc:240] Negotiation successful
0501 20:00:05.150046 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":163,"threads_started":1}
I20240501 20:00:05.152289 21043 heartbeater.cc:347] Connected to a master server at 127.8.168.126:41213
I20240501 20:00:05.152544 21043 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:05.153185 21043 heartbeater.cc:511] Master 127.8.168.126:41213 requested a full tablet report, sending...
I20240501 20:00:05.154356 20558 ts_manager.cc:194] Registered new tserver with Master: 587e967d14484130823c3cba9f68e5d5 (127.8.168.67:46227)
I20240501 20:00:05.155150 20558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:42833
I20240501 20:00:05.159075 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 20:00:05.159674 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 20639
I20240501 20:00:05.165004 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 20779
I20240501 20:00:05.172747 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 20914
I20240501 20:00:05.180851 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 20525
I20240501 20:00:05.187996 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:41213
--webserver_interface=127.8.168.126
--webserver_port=37555
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:41213
--ranger_config_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.168.84:49677/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.168.84:49677/kms with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:05.295200 21046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:05.295460 21046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:05.295511 21046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:05.298616 21046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:00:05.298691 21046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:05.298712 21046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:05.298729 21046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:00:05.298745 21046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:00:05.303350 21046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.168.84:49677/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:41213
--ranger_config_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:41213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=37555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:05.304262 21046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:05.305171 21046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:05.313191 21054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:05.313743 21052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:05.313856 21051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:05.314395 21046 server_base.cc:1034] running on GCE node
I20240501 20:00:05.314900 21046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:05.315850 21046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:05.317004 21046 hybrid_clock.cc:648] HybridClock initialized: now 1714593605316990 us; error 36 us; skew 500 ppm
May 01 20:00:05 dist-test-slave-tzct krb5kdc[20077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593605, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:05.319880 21046 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:00:05.320899 21046 webserver.cc:460] Webserver started at http://127.8.168.126:37555/ using document root <none> and password file <none>
I20240501 20:00:05.321445 21046 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:05.321513 21046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 01 20:00:05 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593605, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for HTTP/127.8.168.84@KRBTEST.COM
I20240501 20:00:05.367427 21046 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:00:05.369597 21062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:05.370721 21046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:05.370862 21046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "914b896f08a645f8ad0fa00ac139f6a8"
format_stamp: "Formatted at 2024-05-01 20:00:03 on dist-test-slave-tzct"
server_key: "0a8a3fe2a5733b9557aaef30dda1ecf8"
server_key_iv: "2551c9517cd5c582033da51748845734"
server_key_version: "kuduclusterkey@0"
I20240501 20:00:05.371305 21046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:05.394541 21046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:05.397778 21046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:05.397976 21046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:05.405316 21046 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:41213
I20240501 20:00:05.405386 21115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:41213 every 8 connection(s)
I20240501 20:00:05.406447 21046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:00:05.410353 21116 sys_catalog.cc:263] Verifying existing consensus state
I20240501 20:00:05.411877 21116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8: Bootstrap starting.
I20240501 20:00:05.414733 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 21046
I20240501 20:00:05.415179 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.168.84:49677/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:34291
--local_ip_for_outbound_sockets=127.8.168.65
--tserver_master_addrs=127.8.168.126:41213
--webserver_port=36535
--webserver_interface=127.8.168.65
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.168.84:49677/kms with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 20:00:05.428261 21116 log.cc:824] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:05.434875 21116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:00:05.435269 21116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8: Bootstrap complete.
I20240501 20:00:05.438750 21116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914b896f08a645f8ad0fa00ac139f6a8" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41213 } }
I20240501 20:00:05.439078 21116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914b896f08a645f8ad0fa00ac139f6a8, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.439551 21116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914b896f08a645f8ad0fa00ac139f6a8" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41213 } }
I20240501 20:00:05.439671 21116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:00:05.439842 21116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:00:05.439955 21116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:05.441676 21116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914b896f08a645f8ad0fa00ac139f6a8" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41213 } }
I20240501 20:00:05.441972 21116 leader_election.cc:302] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 914b896f08a645f8ad0fa00ac139f6a8; no voters:
I20240501 20:00:05.442256 21116 leader_election.cc:288] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240501 20:00:05.442374 21120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:05.442552 21120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [term 2 LEADER]: Becoming Leader. State: Replica: 914b896f08a645f8ad0fa00ac139f6a8, State: Running, Role: LEADER
I20240501 20:00:05.442915 21120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914b896f08a645f8ad0fa00ac139f6a8" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41213 } }
I20240501 20:00:05.443322 21116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:00:05.443799 21120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 914b896f08a645f8ad0fa00ac139f6a8. Latest consensus state: current_term: 2 leader_uuid: "914b896f08a645f8ad0fa00ac139f6a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914b896f08a645f8ad0fa00ac139f6a8" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41213 } } }
I20240501 20:00:05.443974 21120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:05.444108 21122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "914b896f08a645f8ad0fa00ac139f6a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914b896f08a645f8ad0fa00ac139f6a8" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41213 } } }
I20240501 20:00:05.444212 21122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:05.444352 21125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:00:05.447355 21125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:00:05.447858 21116 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20240501 20:00:05.448021 21125 catalog_manager.cc:1259] Loaded cluster ID: 1ea0985aed5a40faaf07f35ca63dd745
I20240501 20:00:05.448067 21125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:00:05.450461 21125 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:00:05.451124 21125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8: Loaded TSK: 0
I20240501 20:00:05.452276 21125 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240501 20:00:05.529281 21118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:05.529588 21118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:05.529721 21118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:05.534243 21118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:05.534328 21118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:05.534443 21118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 20:00:05.540475 21118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.168.84:49677/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:34291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=36535
--tserver_master_addrs=127.8.168.126:41213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:05.541779 21118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:05.542984 21118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:05.553638 21156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:05.553668 21153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:05.554210 21154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:05.554637 21118 server_base.cc:1034] running on GCE node
I20240501 20:00:05.555086 21118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:05.555730 21118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:05.560906 21118 hybrid_clock.cc:648] HybridClock initialized: now 1714593605560882 us; error 41 us; skew 500 ppm
May 01 20:00:05 dist-test-slave-tzct krb5kdc[20077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593605, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:05.564504 21118 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 20:00:05.565651 21118 webserver.cc:460] Webserver started at http://127.8.168.65:36535/ using document root <none> and password file <none>
I20240501 20:00:05.566315 21118 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:05.566404 21118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 01 20:00:05 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593605, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for HTTP/127.8.168.84@KRBTEST.COM
I20240501 20:00:05.595778 21118 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:00:05.598495 21164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:05.599835 21118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:00:05.599967 21118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "289bb9c0d3884c649a1814ed760500bf"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-tzct"
server_key: "52d40152479e9971be11ec1d2e442811"
server_key_iv: "257912a6703ba1ec51d5f70020923752"
server_key_version: "kuduclusterkey@0"
I20240501 20:00:05.600580 21118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:05.619534 21118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:05.623314 21118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:05.623553 21118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:05.624233 21118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:05.625267 21118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:05.625376 21118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:05.625505 21118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:05.625587 21118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:05.636350 21118 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:34291
I20240501 20:00:05.638046 21118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 20:00:05.640702 21277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:34291 every 8 connection(s)
I20240501 20:00:05.646288 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 21118
I20240501 20:00:05.646824 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.168.84:49677/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:35427
--local_ip_for_outbound_sockets=127.8.168.66
--tserver_master_addrs=127.8.168.126:41213
--webserver_port=46881
--webserver_interface=127.8.168.66
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.168.84:49677/kms with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 20:00:05 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593605, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:05.663401 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:05.642859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41213 (local address 127.8.168.65:54867)
0501 20:00:05.643435 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:05.643477 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:05.644453 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:05.646473 (+ 2020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:05.646482 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:05.646749 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:05.647353 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:05.647368 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:05.648518 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:05.648521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:05.649099 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:05.649109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:05.649335 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:05.655774 (+ 6439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:05.655833 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:05.657465 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:05.660109 (+ 2644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:05.660115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:05.660127 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:05.660468 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:05.660859 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:05.660862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:05.660864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:05.660990 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:05.661738 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:05.661745 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:05.662043 (+ 298us) client_negotiation.cc:769] Sending connection context
0501 20:00:05.662276 (+ 233us) client_negotiation.cc:240] Negotiation successful
0501 20:00:05.662562 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":113,"threads_started":1}
I20240501 20:00:05.664973 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:05.642793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:54867 (local address 127.8.168.126:41213)
0501 20:00:05.645002 (+ 2209us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:05.645017 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:05.645061 (+ 44us) server_negotiation.cc:407] Connection header received
0501 20:00:05.645334 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:05.645348 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:05.645712 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:05.646062 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:05.647509 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:05.648364 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:05.656542 (+ 8178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:05.656956 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:05.657685 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:05.657717 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:05.657735 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:05.657782 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:05.659938 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:05.660616 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:05.660622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:05.660627 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:05.660744 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:05.661101 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:05.661105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:05.661107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:05.661540 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:05.661678 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:05.663922 (+ 2244us) server_negotiation.cc:299] Negotiation successful
0501 20:00:05.664194 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2045,"thread_start_us":121,"threads_started":1}
I20240501 20:00:05.668417 21278 heartbeater.cc:347] Connected to a master server at 127.8.168.126:41213
I20240501 20:00:05.668640 21278 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:05.669278 21278 heartbeater.cc:511] Master 127.8.168.126:41213 requested a full tablet report, sending...
W20240501 20:00:05.671365 21278 heartbeater.cc:644] Failed to heartbeat to 127.8.168.126:41213 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20240501 20:00:05.671478 21278 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:05.671696 21278 heartbeater.cc:511] Master 127.8.168.126:41213 requested a full tablet report, sending...
I20240501 20:00:05.672674 21278 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:05.672888 21278 heartbeater.cc:511] Master 127.8.168.126:41213 requested a full tablet report, sending...
W20240501 20:00:05.673804 21278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240501 20:00:05.801811 21282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:05.802079 21282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:05.802184 21282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:05.805680 21282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:05.805757 21282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:05.805861 21282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 20:00:05.810292 21282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.168.84:49677/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:35427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=46881
--tserver_master_addrs=127.8.168.126:41213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:05.811245 21282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:05.812104 21282 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:05.821277 21288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:05.821369 21290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:05.821816 21287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:05.822561 21282 server_base.cc:1034] running on GCE node
I20240501 20:00:05.823011 21282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:05.823773 21282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:05.827083 21282 hybrid_clock.cc:648] HybridClock initialized: now 1714593605826203 us; error 897 us; skew 500 ppm
May 01 20:00:05 dist-test-slave-tzct krb5kdc[20077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593605, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:05.830150 21282 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 20:00:05.831400 21282 webserver.cc:460] Webserver started at http://127.8.168.66:46881/ using document root <none> and password file <none>
I20240501 20:00:05.832098 21282 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:05.832193 21282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 01 20:00:05 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593605, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for HTTP/127.8.168.84@KRBTEST.COM
I20240501 20:00:05.850293 21282 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:00:05.852957 21298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:05.854326 21282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:00:05.854480 21282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "0addd926561444b5943eb0293fbd8eda"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-tzct"
server_key: "44d57f028ae390155e9c9ab241440705"
server_key_iv: "b8f6138babe34d964cfb763a525c46e4"
server_key_version: "kuduclusterkey@0"
I20240501 20:00:05.855091 21282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:05.894567 21282 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:05.898288 21282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:05.898545 21282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:05.899252 21282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:05.900269 21282 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:05.900381 21282 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:05.900485 21282 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:05.900552 21282 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:05.911569 21282 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:35427
I20240501 20:00:05.911964 21411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:35427 every 8 connection(s)
I20240501 20:00:05.913113 21282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 20:00:05.916643 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 21282
I20240501 20:00:05.917193 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.168.84:49677/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:46227
--local_ip_for_outbound_sockets=127.8.168.67
--tserver_master_addrs=127.8.168.126:41213
--webserver_port=41659
--webserver_interface=127.8.168.67
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.168.84:49677/kms with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 20:00:05 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593605, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:05.926836 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:05.914993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:44221 (local address 127.8.168.126:41213)
0501 20:00:05.915120 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:05.915123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:05.916068 (+ 945us) server_negotiation.cc:407] Connection header received
0501 20:00:05.916947 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:05.916950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:05.917012 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:05.917088 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:05.918303 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:05.918951 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:05.919807 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:05.920013 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:05.922515 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:05.922540 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:05.922544 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:05.922575 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:05.924332 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:05.925047 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:05.925051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:05.925053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:05.925112 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:05.925466 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:05.925470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:05.925472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:05.925674 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:05.925772 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:05.926490 (+ 718us) server_negotiation.cc:299] Negotiation successful
0501 20:00:05.926634 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240501 20:00:05.928444 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:05.915420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41213 (local address 127.8.168.66:44221)
0501 20:00:05.915916 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:05.915952 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:05.916691 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:05.917257 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:05.917267 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:05.917592 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:05.918153 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:05.918166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:05.919091 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:05.919096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:05.919671 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:05.919682 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:05.919933 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:05.920650 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:05.920684 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:05.922312 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:05.924485 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:05.924492 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:05.924508 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:05.924917 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:05.925223 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:05.925226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:05.925228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:05.925358 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:05.925841 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:05.925847 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:05.926172 (+ 325us) client_negotiation.cc:769] Sending connection context
0501 20:00:05.927334 (+ 1162us) client_negotiation.cc:240] Negotiation successful
0501 20:00:05.927615 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":140,"threads_started":1}
I20240501 20:00:05.929958 21412 heartbeater.cc:347] Connected to a master server at 127.8.168.126:41213
I20240501 20:00:05.930256 21412 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:05.930981 21412 heartbeater.cc:511] Master 127.8.168.126:41213 requested a full tablet report, sending...
W20240501 20:00:05.932622 21412 heartbeater.cc:644] Failed to heartbeat to 127.8.168.126:41213 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20240501 20:00:05.932840 21412 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:05.933094 21412 heartbeater.cc:511] Master 127.8.168.126:41213 requested a full tablet report, sending...
I20240501 20:00:05.933934 21412 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:05.934165 21412 heartbeater.cc:511] Master 127.8.168.126:41213 requested a full tablet report, sending...
W20240501 20:00:05.934937 21412 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240501 20:00:06.044780 21415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:06.045020 21415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:06.045092 21415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:06.048240 21415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:06.048310 21415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:06.048405 21415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 20:00:06.052660 21415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.168.84:49677/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:46227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=41659
--tserver_master_addrs=127.8.168.126:41213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:06.053639 21415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:06.054529 21415 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:06.062952 21421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:06.064095 21415 server_base.cc:1034] running on GCE node
W20240501 20:00:06.064425 21424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:06.064476 21422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:06.065186 21415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:06.065927 21415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:06.067116 21415 hybrid_clock.cc:648] HybridClock initialized: now 1714593606067086 us; error 39 us; skew 500 ppm
May 01 20:00:06 dist-test-slave-tzct krb5kdc[20077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593606, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:06.070533 21415 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 20:00:06.071697 21415 webserver.cc:460] Webserver started at http://127.8.168.67:41659/ using document root <none> and password file <none>
I20240501 20:00:06.072402 21415 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:06.072505 21415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 01 20:00:06 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593606, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for HTTP/127.8.168.84@KRBTEST.COM
I20240501 20:00:06.106022 21415 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 20:00:06.108943 21432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:06.110363 21415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:00:06.110576 21415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "587e967d14484130823c3cba9f68e5d5"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-tzct"
server_key: "921cc37201878d0de338672381b9019a"
server_key_iv: "6a356041447ee44daaf7786348ba0ea4"
server_key_version: "kuduclusterkey@0"
I20240501 20:00:06.111508 21415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:06.143357 21415 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:06.146427 21415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:06.146593 21415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:06.147127 21415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:06.147898 21415 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:06.147944 21415 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:06.147995 21415 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:06.148023 21415 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:06.157444 21415 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:46227
I20240501 20:00:06.157752 21545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:46227 every 8 connection(s)
I20240501 20:00:06.158453 21415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 20:00:06.165131 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 21415
May 01 20:00:06 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593606, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:06.174548 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:06.161233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:34345 (local address 127.8.168.126:41213)
0501 20:00:06.161373 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:06.161377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:06.162510 (+ 1133us) server_negotiation.cc:407] Connection header received
0501 20:00:06.163549 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:06.163552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:06.163602 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:06.163678 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:06.165294 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:06.166071 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:06.167707 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:06.167916 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:06.170499 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:06.170519 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:06.170522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:06.170548 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:06.172279 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:06.172901 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:06.172905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:06.172907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:06.172958 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:06.173307 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:06.173311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:06.173313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:06.173481 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:06.173580 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:06.174238 (+ 658us) server_negotiation.cc:299] Negotiation successful
0501 20:00:06.174358 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240501 20:00:06.175791 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:06.161590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41213 (local address 127.8.168.67:34345)
0501 20:00:06.162354 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:06.162399 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:06.163315 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:06.163891 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:06.163902 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:06.164241 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:06.165063 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:06.165139 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:06.166991 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:06.166994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:06.167572 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:06.167583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:06.168154 (+ 571us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:06.168860 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:06.168895 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:06.170333 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:06.172397 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:06.172403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:06.172415 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:06.172777 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:06.173062 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:06.173065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:06.173067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:06.173181 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:06.173599 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:06.173605 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:06.173937 (+ 332us) client_negotiation.cc:769] Sending connection context
0501 20:00:06.174165 (+ 228us) client_negotiation.cc:240] Negotiation successful
0501 20:00:06.174427 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":99,"threads_started":1}
I20240501 20:00:06.177086 21546 heartbeater.cc:347] Connected to a master server at 127.8.168.126:41213
I20240501 20:00:06.177331 21546 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:06.177943 21546 heartbeater.cc:511] Master 127.8.168.126:41213 requested a full tablet report, sending...
W20240501 20:00:06.179390 21546 heartbeater.cc:644] Failed to heartbeat to 127.8.168.126:41213 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20240501 20:00:06.179563 21546 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:06.179772 21546 heartbeater.cc:511] Master 127.8.168.126:41213 requested a full tablet report, sending...
I20240501 20:00:06.180557 21546 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:06.180778 21546 heartbeater.cc:511] Master 127.8.168.126:41213 requested a full tablet report, sending...
I20240501 20:00:06.180939 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:06.166484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46634 (local address 127.8.168.126:41213)
0501 20:00:06.169705 (+ 3221us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:06.169709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:06.169721 (+ 12us) server_negotiation.cc:407] Connection header received
0501 20:00:06.169765 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:06.169768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:06.169816 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:06.169914 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:06.170862 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:06.172798 (+ 1936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:06.174797 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:06.174984 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:06.176365 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:06.176382 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:06.176384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:06.176404 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:06.179147 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:06.179569 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:06.179573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:06.179575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:06.179620 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:06.179920 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:06.179924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:06.179925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:06.180089 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:06.180252 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:06.180501 (+ 249us) server_negotiation.cc:299] Negotiation successful
0501 20:00:06.180767 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3137,"thread_start_us":81,"threads_started":1}
W20240501 20:00:06.181617 21546 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
May 01 20:00:06 dist-test-slave-tzct krb5kdc[20077](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1714593606, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:06.674031 21278 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:06.674389 21278 heartbeater.cc:511] Master 127.8.168.126:41213 requested a full tablet report, sending...
I20240501 20:00:06.675540 21080 ts_manager.cc:194] Registered new tserver with Master: 289bb9c0d3884c649a1814ed760500bf (127.8.168.65:34291)
I20240501 20:00:06.677404 21080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:54867
May 01 20:00:06 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593606, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for HTTP/127.8.168.84@KRBTEST.COM
I20240501 20:00:06.935184 21412 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:06.935508 21412 heartbeater.cc:511] Master 127.8.168.126:41213 requested a full tablet report, sending...
I20240501 20:00:06.936336 21080 ts_manager.cc:194] Registered new tserver with Master: 0addd926561444b5943eb0293fbd8eda (127.8.168.66:35427)
I20240501 20:00:06.937134 21080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:44221
I20240501 20:00:07.181890 21546 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:07.182240 21546 heartbeater.cc:511] Master 127.8.168.126:41213 requested a full tablet report, sending...
I20240501 20:00:07.183076 21080 ts_manager.cc:194] Registered new tserver with Master: 587e967d14484130823c3cba9f68e5d5 (127.8.168.67:46227)
I20240501 20:00:07.183873 21080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:34345
I20240501 20:00:07.194310 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 01 20:00:07 dist-test-slave-tzct krb5kdc[20077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593606, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for HTTP/127.8.168.84@KRBTEST.COM
I20240501 20:00:07.211400 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:07.197060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46638 (local address 127.8.168.126:41213)
0501 20:00:07.197683 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:07.197688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:07.197711 (+ 23us) server_negotiation.cc:407] Connection header received
0501 20:00:07.197750 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:07.197754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:07.197806 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:07.197899 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:07.199002 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.199855 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:07.206256 (+ 6401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.206484 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:07.207710 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:07.207733 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:07.207735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:07.207762 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:07.209579 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:07.210042 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:07.210045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:07.210047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:07.210102 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:07.210431 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:07.210435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:07.210436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:07.210657 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:07.210763 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:07.211053 (+ 290us) server_negotiation.cc:299] Negotiation successful
0501 20:00:07.211166 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":76,"threads_started":1}
I20240501 20:00:07.215283 21080 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:00:07.219655 21080 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:00:07.238031 21574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:07.233058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:34291 (local address 127.0.0.1:34012)
0501 20:00:07.233746 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:07.233794 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:07.233943 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:07.234883 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:07.234889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:07.234914 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:07.235211 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:07.235222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.236574 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:07.236578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:07.237481 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:07.237494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.237603 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:07.237664 (+ 61us) client_negotiation.cc:769] Sending connection context
0501 20:00:07.237762 (+ 98us) client_negotiation.cc:240] Negotiation successful
0501 20:00:07.237875 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":100,"threads_started":1}
I20240501 20:00:07.239737 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:07.233311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34012 (local address 127.8.168.65:34291)
0501 20:00:07.234225 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:07.234232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:07.234256 (+ 24us) server_negotiation.cc:407] Connection header received
0501 20:00:07.234339 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:07.234344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:07.234523 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:07.234679 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:07.235349 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.236447 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:07.238752 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.239278 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:07.239406 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:07.239484 (+ 78us) server_negotiation.cc:299] Negotiation successful
0501 20:00:07.239571 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":108,"threads_started":1}
I20240501 20:00:07.242477 21574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:07.233775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:46227 (local address 127.0.0.1:45188)
0501 20:00:07.238550 (+ 4775us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:07.238563 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:07.238633 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:07.239646 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:07.239649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:07.239665 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:07.239940 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:07.239951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.241288 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:07.241291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:07.242111 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:07.242122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.242226 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:07.242244 (+ 18us) client_negotiation.cc:769] Sending connection context
0501 20:00:07.242298 (+ 54us) client_negotiation.cc:240] Negotiation successful
0501 20:00:07.242354 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4715,"thread_start_us":90,"threads_started":1}
I20240501 20:00:07.242573 21212 tablet_service.cc:1467] Processing CreateTablet for tablet c994804844fc464f98d89d60f8d46632 (DEFAULT_TABLE table=test-table [id=5f0fb4eb5f5c4b1ab0203748fb01fab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:07.243613 21212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c994804844fc464f98d89d60f8d46632. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:07.243719 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:07.233987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45188 (local address 127.8.168.67:46227)
0501 20:00:07.238671 (+ 4684us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:07.238678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:07.238702 (+ 24us) server_negotiation.cc:407] Connection header received
0501 20:00:07.238796 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:07.238802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:07.238988 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:07.239140 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:07.240080 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.241151 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:07.242721 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.243251 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:07.243380 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:07.243456 (+ 76us) server_negotiation.cc:299] Negotiation successful
0501 20:00:07.243530 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4575,"thread_start_us":88,"threads_started":1}
I20240501 20:00:07.246598 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:07.234108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:35427 (local address 127.0.0.1:44268)
0501 20:00:07.239387 (+ 5279us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:07.239401 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:07.239476 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:07.243707 (+ 4231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:07.243710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:07.243729 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:07.244017 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:07.244027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.245343 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:07.245347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:07.246216 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:07.246228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.246337 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:07.246356 (+ 19us) client_negotiation.cc:769] Sending connection context
0501 20:00:07.246407 (+ 51us) client_negotiation.cc:240] Negotiation successful
0501 20:00:07.246463 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5210,"thread_start_us":78,"threads_started":1}
I20240501 20:00:07.246438 21480 tablet_service.cc:1467] Processing CreateTablet for tablet c994804844fc464f98d89d60f8d46632 (DEFAULT_TABLE table=test-table [id=5f0fb4eb5f5c4b1ab0203748fb01fab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:07.247840 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:07.234276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44268 (local address 127.8.168.66:35427)
0501 20:00:07.243104 (+ 8828us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:07.243110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:07.243136 (+ 26us) server_negotiation.cc:407] Connection header received
0501 20:00:07.243230 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:07.243235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:07.243409 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:07.243566 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:07.244154 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.245225 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:07.246888 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.247419 (+ 531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:07.247536 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:07.247606 (+ 70us) server_negotiation.cc:299] Negotiation successful
0501 20:00:07.247682 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8711,"thread_start_us":94,"threads_started":1}
I20240501 20:00:07.250465 21339 tablet_service.cc:1467] Processing CreateTablet for tablet c994804844fc464f98d89d60f8d46632 (DEFAULT_TABLE table=test-table [id=5f0fb4eb5f5c4b1ab0203748fb01fab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:07.250931 21480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c994804844fc464f98d89d60f8d46632. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:07.251425 21339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c994804844fc464f98d89d60f8d46632. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:07.259567 21581 tablet_bootstrap.cc:492] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda: Bootstrap starting.
I20240501 20:00:07.262212 21581 tablet_bootstrap.cc:654] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:07.263051 21581 log.cc:824] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:07.263806 21582 tablet_bootstrap.cc:492] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5: Bootstrap starting.
I20240501 20:00:07.265781 21582 tablet_bootstrap.cc:654] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:07.266158 21583 tablet_bootstrap.cc:492] T c994804844fc464f98d89d60f8d46632 P 289bb9c0d3884c649a1814ed760500bf: Bootstrap starting.
I20240501 20:00:07.266605 21582 log.cc:824] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:07.267444 21581 tablet_bootstrap.cc:492] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda: No bootstrap required, opened a new log
I20240501 20:00:07.267684 21581 ts_tablet_manager.cc:1367] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 20:00:07.268422 21582 tablet_bootstrap.cc:492] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5: No bootstrap required, opened a new log
I20240501 20:00:07.268585 21582 ts_tablet_manager.cc:1367] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 20:00:07.268761 21583 tablet_bootstrap.cc:654] T c994804844fc464f98d89d60f8d46632 P 289bb9c0d3884c649a1814ed760500bf: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:07.270102 21583 log.cc:824] T c994804844fc464f98d89d60f8d46632 P 289bb9c0d3884c649a1814ed760500bf: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:07.271272 21581 raft_consensus.cc:348] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289bb9c0d3884c649a1814ed760500bf" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34291 } } peers { permanent_uuid: "587e967d14484130823c3cba9f68e5d5" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 46227 } } peers { permanent_uuid: "0addd926561444b5943eb0293fbd8eda" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35427 } }
I20240501 20:00:07.271505 21581 raft_consensus.cc:374] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:07.271555 21581 raft_consensus.cc:729] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0addd926561444b5943eb0293fbd8eda, State: Initialized, Role: FOLLOWER
I20240501 20:00:07.271948 21583 tablet_bootstrap.cc:492] T c994804844fc464f98d89d60f8d46632 P 289bb9c0d3884c649a1814ed760500bf: No bootstrap required, opened a new log
I20240501 20:00:07.271972 21582 raft_consensus.cc:348] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289bb9c0d3884c649a1814ed760500bf" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34291 } } peers { permanent_uuid: "587e967d14484130823c3cba9f68e5d5" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 46227 } } peers { permanent_uuid: "0addd926561444b5943eb0293fbd8eda" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35427 } }
I20240501 20:00:07.272001 21581 consensus_queue.cc:260] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289bb9c0d3884c649a1814ed760500bf" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34291 } } peers { permanent_uuid: "587e967d14484130823c3cba9f68e5d5" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 46227 } } peers { permanent_uuid: "0addd926561444b5943eb0293fbd8eda" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35427 } }
I20240501 20:00:07.272118 21583 ts_tablet_manager.cc:1367] T c994804844fc464f98d89d60f8d46632 P 289bb9c0d3884c649a1814ed760500bf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:00:07.272168 21582 raft_consensus.cc:374] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:07.272224 21582 raft_consensus.cc:729] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587e967d14484130823c3cba9f68e5d5, State: Initialized, Role: FOLLOWER
I20240501 20:00:07.272691 21582 consensus_queue.cc:260] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289bb9c0d3884c649a1814ed760500bf" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34291 } } peers { permanent_uuid: "587e967d14484130823c3cba9f68e5d5" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 46227 } } peers { permanent_uuid: "0addd926561444b5943eb0293fbd8eda" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35427 } }
I20240501 20:00:07.272972 21581 ts_tablet_manager.cc:1398] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 20:00:07.273370 21412 heartbeater.cc:503] Master 127.8.168.126:41213 was elected leader, sending a full tablet report...
I20240501 20:00:07.273566 21582 ts_tablet_manager.cc:1398] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 20:00:07.273895 21546 heartbeater.cc:503] Master 127.8.168.126:41213 was elected leader, sending a full tablet report...
I20240501 20:00:07.276211 21583 raft_consensus.cc:348] T c994804844fc464f98d89d60f8d46632 P 289bb9c0d3884c649a1814ed760500bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289bb9c0d3884c649a1814ed760500bf" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34291 } } peers { permanent_uuid: "587e967d14484130823c3cba9f68e5d5" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 46227 } } peers { permanent_uuid: "0addd926561444b5943eb0293fbd8eda" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35427 } }
I20240501 20:00:07.276412 21583 raft_consensus.cc:374] T c994804844fc464f98d89d60f8d46632 P 289bb9c0d3884c649a1814ed760500bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:07.276477 21583 raft_consensus.cc:729] T c994804844fc464f98d89d60f8d46632 P 289bb9c0d3884c649a1814ed760500bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289bb9c0d3884c649a1814ed760500bf, State: Initialized, Role: FOLLOWER
I20240501 20:00:07.276929 21583 consensus_queue.cc:260] T c994804844fc464f98d89d60f8d46632 P 289bb9c0d3884c649a1814ed760500bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289bb9c0d3884c649a1814ed760500bf" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34291 } } peers { permanent_uuid: "587e967d14484130823c3cba9f68e5d5" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 46227 } } peers { permanent_uuid: "0addd926561444b5943eb0293fbd8eda" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35427 } }
I20240501 20:00:07.277812 21583 ts_tablet_manager.cc:1398] T c994804844fc464f98d89d60f8d46632 P 289bb9c0d3884c649a1814ed760500bf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 20:00:07.278178 21278 heartbeater.cc:503] Master 127.8.168.126:41213 was elected leader, sending a full tablet report...
W20240501 20:00:07.393761 21279 tablet.cc:2369] T c994804844fc464f98d89d60f8d46632 P 289bb9c0d3884c649a1814ed760500bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:00:07.410113 21547 tablet.cc:2369] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:00:07.414140 21588 raft_consensus.cc:482] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:07.414356 21588 raft_consensus.cc:504] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289bb9c0d3884c649a1814ed760500bf" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34291 } } peers { permanent_uuid: "587e967d14484130823c3cba9f68e5d5" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 46227 } } peers { permanent_uuid: "0addd926561444b5943eb0293fbd8eda" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35427 } }
I20240501 20:00:07.415725 21588 leader_election.cc:288] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 289bb9c0d3884c649a1814ed760500bf (127.8.168.65:34291), 0addd926561444b5943eb0293fbd8eda (127.8.168.66:35427)
W20240501 20:00:07.416955 21413 tablet.cc:2369] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:00:07.421370 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:07.417624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:59271 (local address 127.8.168.66:35427)
0501 20:00:07.417760 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:07.417764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:07.417777 (+ 13us) server_negotiation.cc:407] Connection header received
0501 20:00:07.417923 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:07.417927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:07.417972 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:07.418021 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:07.418491 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.419210 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:07.420416 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.420905 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:07.420949 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:07.421163 (+ 214us) server_negotiation.cc:299] Negotiation successful
0501 20:00:07.421245 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20240501 20:00:07.421370 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:07.416864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:53869 (local address 127.8.168.65:34291)
0501 20:00:07.416995 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:07.416999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:07.417368 (+ 369us) server_negotiation.cc:407] Connection header received
0501 20:00:07.417590 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:07.417594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:07.417651 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:07.417740 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:07.418400 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.419202 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:07.420497 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.420990 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:07.421023 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:07.421190 (+ 167us) server_negotiation.cc:299] Negotiation successful
0501 20:00:07.421248 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240501 20:00:07.421705 21590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:07.416411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:35427 (local address 127.8.168.67:59271)
0501 20:00:07.417706 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:07.417718 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:07.417801 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:07.418129 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:07.418132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:07.418147 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:07.418372 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:07.418380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.419345 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:07.419348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:07.420279 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:07.420290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.421013 (+ 723us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:07.421030 (+ 17us) client_negotiation.cc:769] Sending connection context
0501 20:00:07.421499 (+ 469us) client_negotiation.cc:240] Negotiation successful
0501 20:00:07.421567 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1230,"thread_start_us":98,"threads_started":1}
I20240501 20:00:07.422273 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:07.415965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:34291 (local address 127.8.168.67:53869)
0501 20:00:07.417285 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:07.417300 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:07.417451 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:07.417891 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:07.417894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:07.417911 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:07.418223 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:07.418231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.419394 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:07.419397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:07.420383 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:07.420391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.421068 (+ 677us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:07.421080 (+ 12us) client_negotiation.cc:769] Sending connection context
0501 20:00:07.422083 (+ 1003us) client_negotiation.cc:240] Negotiation successful
0501 20:00:07.422144 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1198,"thread_start_us":118,"threads_started":1}
I20240501 20:00:07.422420 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c994804844fc464f98d89d60f8d46632" candidate_uuid: "587e967d14484130823c3cba9f68e5d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0addd926561444b5943eb0293fbd8eda" is_pre_election: true
I20240501 20:00:07.422726 21356 raft_consensus.cc:2454] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 587e967d14484130823c3cba9f68e5d5 in term 0.
I20240501 20:00:07.423004 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c994804844fc464f98d89d60f8d46632" candidate_uuid: "587e967d14484130823c3cba9f68e5d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289bb9c0d3884c649a1814ed760500bf" is_pre_election: true
I20240501 20:00:07.423233 21228 raft_consensus.cc:2454] T c994804844fc464f98d89d60f8d46632 P 289bb9c0d3884c649a1814ed760500bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 587e967d14484130823c3cba9f68e5d5 in term 0.
I20240501 20:00:07.423660 21434 leader_election.cc:302] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289bb9c0d3884c649a1814ed760500bf, 587e967d14484130823c3cba9f68e5d5; no voters:
I20240501 20:00:07.423933 21588 raft_consensus.cc:2781] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:07.424078 21588 raft_consensus.cc:482] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:07.424229 21588 raft_consensus.cc:3037] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:07.427942 21588 raft_consensus.cc:504] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289bb9c0d3884c649a1814ed760500bf" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34291 } } peers { permanent_uuid: "587e967d14484130823c3cba9f68e5d5" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 46227 } } peers { permanent_uuid: "0addd926561444b5943eb0293fbd8eda" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35427 } }
I20240501 20:00:07.429500 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c994804844fc464f98d89d60f8d46632" candidate_uuid: "587e967d14484130823c3cba9f68e5d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "289bb9c0d3884c649a1814ed760500bf"
I20240501 20:00:07.429636 21228 raft_consensus.cc:3037] T c994804844fc464f98d89d60f8d46632 P 289bb9c0d3884c649a1814ed760500bf [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:07.430830 21228 raft_consensus.cc:2454] T c994804844fc464f98d89d60f8d46632 P 289bb9c0d3884c649a1814ed760500bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 587e967d14484130823c3cba9f68e5d5 in term 1.
I20240501 20:00:07.431196 21434 leader_election.cc:302] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289bb9c0d3884c649a1814ed760500bf, 587e967d14484130823c3cba9f68e5d5; no voters:
I20240501 20:00:07.431878 21592 raft_consensus.cc:2781] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:07.431915 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c994804844fc464f98d89d60f8d46632" candidate_uuid: "587e967d14484130823c3cba9f68e5d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0addd926561444b5943eb0293fbd8eda"
I20240501 20:00:07.432049 21356 raft_consensus.cc:3037] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:07.432202 21592 raft_consensus.cc:686] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [term 1 LEADER]: Becoming Leader. State: Replica: 587e967d14484130823c3cba9f68e5d5, State: Running, Role: LEADER
I20240501 20:00:07.432597 21592 consensus_queue.cc:237] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289bb9c0d3884c649a1814ed760500bf" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34291 } } peers { permanent_uuid: "587e967d14484130823c3cba9f68e5d5" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 46227 } } peers { permanent_uuid: "0addd926561444b5943eb0293fbd8eda" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35427 } }
I20240501 20:00:07.433272 21356 raft_consensus.cc:2454] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 587e967d14484130823c3cba9f68e5d5 in term 1.
I20240501 20:00:07.433642 21588 leader_election.cc:288] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [CANDIDATE]: Term 1 election: Requested vote from peers 289bb9c0d3884c649a1814ed760500bf (127.8.168.65:34291), 0addd926561444b5943eb0293fbd8eda (127.8.168.66:35427)
I20240501 20:00:07.436182 21079 catalog_manager.cc:5484] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 587e967d14484130823c3cba9f68e5d5 (127.8.168.67). New cstate: current_term: 1 leader_uuid: "587e967d14484130823c3cba9f68e5d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "289bb9c0d3884c649a1814ed760500bf" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "587e967d14484130823c3cba9f68e5d5" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 46227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0addd926561444b5943eb0293fbd8eda" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35427 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:07.480852 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:07.477965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45204 (local address 127.8.168.67:46227)
0501 20:00:07.478090 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:07.478093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:07.478105 (+ 12us) server_negotiation.cc:407] Connection header received
0501 20:00:07.478181 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:07.478184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:07.478243 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:07.478343 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:00:07.478654 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.479163 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:07.479820 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:07.479971 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:07.480059 (+ 88us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:00:07.480485 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:00:07.480577 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:07.480652 (+ 75us) server_negotiation.cc:299] Negotiation successful
0501 20:00:07.480713 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 20:00:07.486166 21356 raft_consensus.cc:1261] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda [term 1 FOLLOWER]: Refusing update from remote peer 587e967d14484130823c3cba9f68e5d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:00:07.486166 21228 raft_consensus.cc:1261] T c994804844fc464f98d89d60f8d46632 P 289bb9c0d3884c649a1814ed760500bf [term 1 FOLLOWER]: Refusing update from remote peer 587e967d14484130823c3cba9f68e5d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:00:07.486819 21593 consensus_queue.cc:1035] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "289bb9c0d3884c649a1814ed760500bf" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:07.486991 21592 consensus_queue.cc:1035] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0addd926561444b5943eb0293fbd8eda" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:07.495934 21604 mvcc.cc:205] Tried to move back new op lower bound from 7022975416255725568 to 7022975416047988736. Current Snapshot: MvccSnapshot[applied={T|T < 7022975416255725568}]
I20240501 20:00:07.501813 21079 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46638:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:00:07.501986 21079 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46638:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:00:07.504093 21079 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 914b896f08a645f8ad0fa00ac139f6a8: Sending DeleteTablet for 3 replicas of tablet c994804844fc464f98d89d60f8d46632
I20240501 20:00:07.504844 21212 tablet_service.cc:1514] Processing DeleteTablet for tablet c994804844fc464f98d89d60f8d46632 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:00:07 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:34012
I20240501 20:00:07.504884 21480 tablet_service.cc:1514] Processing DeleteTablet for tablet c994804844fc464f98d89d60f8d46632 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:00:07 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:45188
I20240501 20:00:07.504926 21339 tablet_service.cc:1514] Processing DeleteTablet for tablet c994804844fc464f98d89d60f8d46632 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:00:07 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:44268
I20240501 20:00:07.505380 21614 tablet_replica.cc:330] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda: stopping tablet replica
I20240501 20:00:07.505530 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 21118
I20240501 20:00:07.505635 21614 raft_consensus.cc:2229] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:07.506476 21614 raft_consensus.cc:2258] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:07.507992 21614 ts_tablet_manager.cc:1875] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:00:07.510641 21614 ts_tablet_manager.cc:1888] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:00:07.510744 21614 log.cc:1196] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/c994804844fc464f98d89d60f8d46632
I20240501 20:00:07.511034 21612 tablet_replica.cc:330] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5: stopping tablet replica
I20240501 20:00:07.511054 21614 ts_tablet_manager.cc:1909] T c994804844fc464f98d89d60f8d46632 P 0addd926561444b5943eb0293fbd8eda: Deleting consensus metadata
I20240501 20:00:07.511384 21612 raft_consensus.cc:2229] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [term 1 LEADER]: Raft consensus shutting down.
W20240501 20:00:07.511788 21064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.168.65:34291 (error 108)
I20240501 20:00:07.511848 21612 raft_consensus.cc:2258] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:07.512287 21064 catalog_manager.cc:4830] TS 0addd926561444b5943eb0293fbd8eda (127.8.168.66:35427): tablet c994804844fc464f98d89d60f8d46632 (table test-table [id=5f0fb4eb5f5c4b1ab0203748fb01fab0]) successfully deleted
W20240501 20:00:07.513033 21578 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:00:07.512749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:34291 (local address 127.0.0.1:34020)
0501 20:00:07.512881 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:07.512942 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:34291: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":44}
I20240501 20:00:07.513053 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 21282
W20240501 20:00:07.513888 21064 catalog_manager.cc:4573] TS 289bb9c0d3884c649a1814ed760500bf (127.8.168.65:34291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c994804844fc464f98d89d60f8d46632: Network error: Client connection negotiation failed: client connection to 127.8.168.65:34291: connect: Connection reset by peer (error 104)
I20240501 20:00:07.513804 21612 ts_tablet_manager.cc:1875] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:00:07.516582 21612 ts_tablet_manager.cc:1888] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:00:07.516750 21612 log.cc:1196] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/c994804844fc464f98d89d60f8d46632
I20240501 20:00:07.517095 21612 ts_tablet_manager.cc:1909] T c994804844fc464f98d89d60f8d46632 P 587e967d14484130823c3cba9f68e5d5: Deleting consensus metadata
I20240501 20:00:07.518442 21064 catalog_manager.cc:4830] TS 587e967d14484130823c3cba9f68e5d5 (127.8.168.67:46227): tablet c994804844fc464f98d89d60f8d46632 (table test-table [id=5f0fb4eb5f5c4b1ab0203748fb01fab0]) successfully deleted
I20240501 20:00:07.518687 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 21415
I20240501 20:00:07.523407 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 21046
I20240501 20:00:07.529173 8865 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20240501 20:00:07.863963 8865 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20240501 20:00:07.864099 8865 mini_ranger.cc:66] Stopping Ranger...
I20240501 20:00:08.199208 8865 mini_ranger.cc:68] Stopped Ranger
2024-05-01 20:00:08.199 UTC [20107] LOG: received smart shutdown request
2024-05-01 20:00:08.202 UTC [20107] LOG: background worker "logical replication launcher" (PID 20116) exited with exit code 1
2024-05-01 20:00:08.202 UTC [20111] LOG: shutting down
2024-05-01 20:00:08.358 UTC [20107] LOG: database system is shut down
2024-05-01T20:00:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47361 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:00:08 dist-test-slave-tzct krb5kdc[21624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:00:08 dist-test-slave-tzct krb5kdc[21624](info): set up 2 sockets
May 01 20:00:08 dist-test-slave-tzct krb5kdc[21624](info): commencing operation
krb5kdc: starting...
W20240501 20:00:09.344781 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.874s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:00:09 dist-test-slave-tzct krb5kdc[21624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:00:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:00:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 20:00:09.455842 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:45759
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:45759,127.8.168.125:36605,127.8.168.124:33909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:09.560686 21637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:09.560936 21637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:09.560997 21637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:09.564067 21637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:00:09.564129 21637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:09.564152 21637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:09.564169 21637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:00:09.564185 21637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:00:09.568440 21637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:45759,127.8.168.125:36605,127.8.168.124:33909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:45759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:09.569337 21637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:09.570151 21637 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:09.576982 21643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:09.577625 21642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:09.577963 21645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:09.578617 21637 server_base.cc:1034] running on GCE node
I20240501 20:00:09.579070 21637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:09.579958 21637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:09.581107 21637 hybrid_clock.cc:648] HybridClock initialized: now 1714593609581081 us; error 40 us; skew 500 ppm
May 01 20:00:09 dist-test-slave-tzct krb5kdc[21624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:09.583703 21637 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:00:09.584625 21637 webserver.cc:460] Webserver started at http://127.8.168.126:46115/ using document root <none> and password file <none>
I20240501 20:00:09.585166 21637 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:09.585223 21637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:09.585451 21637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:09.587260 21637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "c7df2c6884a543ca84a325c08a2bc11b"
format_stamp: "Formatted at 2024-05-01 20:00:09 on dist-test-slave-tzct"
server_key: "c0dd939c1688a8619bef3d352558be7a"
server_key_iv: "756096bbbddd4d472ad902455611d987"
server_key_version: "encryptionkey@0"
I20240501 20:00:09.587736 21637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "c7df2c6884a543ca84a325c08a2bc11b"
format_stamp: "Formatted at 2024-05-01 20:00:09 on dist-test-slave-tzct"
server_key: "c0dd939c1688a8619bef3d352558be7a"
server_key_iv: "756096bbbddd4d472ad902455611d987"
server_key_version: "encryptionkey@0"
I20240501 20:00:09.591099 21637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 20:00:09.593334 21652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:09.594336 21637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:09.594453 21637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "c7df2c6884a543ca84a325c08a2bc11b"
format_stamp: "Formatted at 2024-05-01 20:00:09 on dist-test-slave-tzct"
server_key: "c0dd939c1688a8619bef3d352558be7a"
server_key_iv: "756096bbbddd4d472ad902455611d987"
server_key_version: "encryptionkey@0"
I20240501 20:00:09.594558 21637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:09.604979 21637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:09.607719 21637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:09.607875 21637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:09.614513 21637 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:45759
I20240501 20:00:09.614531 21704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:45759 every 8 connection(s)
I20240501 20:00:09.615506 21637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:00:09.618319 21705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:09.621287 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 21637
I20240501 20:00:09.621424 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:00:09.621688 8865 external_mini_cluster.cc:1428] Setting key eaf7b9b63ca2824bb1c5171f0f729450
I20240501 20:00:09.621776 21705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.168.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.125@KRBTEST.COM" created.
May 01 20:00:09 dist-test-slave-tzct krb5kdc[21624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:09.637032 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.624112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.8.168.126:45759)
0501 20:00:09.624521 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:09.624531 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:09.624747 (+ 216us) server_negotiation.cc:407] Connection header received
0501 20:00:09.625707 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:09.625711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:09.625866 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:09.626030 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:09.626950 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:09.627631 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:09.628176 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:09.628363 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:09.632227 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:09.632254 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:09.632267 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:09.632295 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:09.634040 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:09.634613 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:09.634618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:09.634619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:09.634687 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:09.634980 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:09.634982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:09.634983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:09.635314 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:09.635427 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:09.635902 (+ 475us) server_negotiation.cc:299] Negotiation successful
0501 20:00:09.636136 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"spinlock_wait_cycles":23808,"thread_start_us":114,"threads_started":1}
I20240501 20:00:09.637037 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.624113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45759 (local address 127.0.0.1:48598)
0501 20:00:09.624600 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.624632 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:09.625420 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:09.626041 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:09.626046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:09.626526 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:09.626816 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:09.626829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:09.627727 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:09.627730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:09.628074 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:09.628081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:09.628292 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:09.629003 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:09.629030 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:09.632066 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:09.634161 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:09.634172 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:09.634181 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:09.634488 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:09.634775 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:09.634779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:09.634781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:09.634891 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:09.635434 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:09.635441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:09.635621 (+ 180us) client_negotiation.cc:769] Sending connection context
0501 20:00:09.635860 (+ 239us) client_negotiation.cc:240] Negotiation successful
0501 20:00:09.636277 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":105,"spinlock_wait_cycles":13056,"thread_start_us":143,"threads_started":1}
I20240501 20:00:09.639475 21705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:00:09.640172 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.639904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36605 (local address 127.0.0.1:43478)
0501 20:00:09.640031 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.640102 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20240501 20:00:09.640349 21653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36605: connect: Connection refused (error 111)
I20240501 20:00:09.641139 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.640944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36605 (local address 127.0.0.1:43486)
0501 20:00:09.641042 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.641083 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20240501 20:00:09.641260 21705 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.125:36605: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36605: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.125.keytab.
Entry for principal kudu/127.8.168.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.125.keytab.
I20240501 20:00:09.646116 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.125:36605
--webserver_interface=127.8.168.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.125.keytab
--principal=kudu/127.8.168.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:45759,127.8.168.125:36605,127.8.168.124:33909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 20:00:09.690480 21705 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } attempt: 1
I20240501 20:00:09.691278 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.691002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36605 (local address 127.0.0.1:43498)
0501 20:00:09.691145 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.691202 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20240501 20:00:09.692008 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.691818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36605 (local address 127.0.0.1:43502)
0501 20:00:09.691906 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.691946 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20240501 20:00:09.692119 21705 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.125:36605: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36605: connect: Connection refused (error 111)
W20240501 20:00:09.749938 21712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:09.750150 21712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:09.750201 21712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:09.753180 21712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:00:09.753239 21712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:09.753261 21712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:09.753288 21712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:00:09.753304 21712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:00:09.757409 21712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:45759,127.8.168.125:36605,127.8.168.124:33909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.125.keytab
--principal=kudu/127.8.168.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.125:36605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.168.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:09.758231 21712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:09.759002 21712 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:00:09.760246 21705 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } attempt: 2
I20240501 20:00:09.761036 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.760775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36605 (local address 127.0.0.1:43506)
0501 20:00:09.760907 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.760960 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20240501 20:00:09.761705 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.761514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36605 (local address 127.0.0.1:43510)
0501 20:00:09.761606 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.761647 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20240501 20:00:09.761824 21705 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.125:36605: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36605: connect: Connection refused (error 111)
W20240501 20:00:09.765931 21720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:09.766494 21717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:09.766847 21718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:09.767482 21712 server_base.cc:1034] running on GCE node
I20240501 20:00:09.767956 21712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:09.768890 21712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:09.770033 21712 hybrid_clock.cc:648] HybridClock initialized: now 1714593609770023 us; error 26 us; skew 500 ppm
May 01 20:00:09 dist-test-slave-tzct krb5kdc[21624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:09.772585 21712 init.cc:377] Logged in from keytab as kudu/127.8.168.125@KRBTEST.COM (short username kudu)
I20240501 20:00:09.773476 21712 webserver.cc:460] Webserver started at http://127.8.168.125:40937/ using document root <none> and password file <none>
I20240501 20:00:09.773993 21712 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:09.774049 21712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:09.774211 21712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:09.775915 21712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-1/data/instance:
uuid: "f9567ba68c2a4af39b43b0f79f3938f1"
format_stamp: "Formatted at 2024-05-01 20:00:09 on dist-test-slave-tzct"
server_key: "f5e0a9499de69350cf872acb6af961aa"
server_key_iv: "2ff43c60d09435b537c743066c2a0c86"
server_key_version: "encryptionkey@0"
I20240501 20:00:09.776373 21712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-1/wal/instance:
uuid: "f9567ba68c2a4af39b43b0f79f3938f1"
format_stamp: "Formatted at 2024-05-01 20:00:09 on dist-test-slave-tzct"
server_key: "f5e0a9499de69350cf872acb6af961aa"
server_key_iv: "2ff43c60d09435b537c743066c2a0c86"
server_key_version: "encryptionkey@0"
I20240501 20:00:09.779700 21712 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20240501 20:00:09.781929 21727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:09.782940 21712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:00:09.783047 21712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-1/wal
uuid: "f9567ba68c2a4af39b43b0f79f3938f1"
format_stamp: "Formatted at 2024-05-01 20:00:09 on dist-test-slave-tzct"
server_key: "f5e0a9499de69350cf872acb6af961aa"
server_key_iv: "2ff43c60d09435b537c743066c2a0c86"
server_key_version: "encryptionkey@0"
I20240501 20:00:09.783138 21712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:09.798077 21712 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:09.801039 21712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:09.801203 21712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:09.807911 21712 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.125:36605
I20240501 20:00:09.807960 21779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.125:36605 every 8 connection(s)
I20240501 20:00:09.808895 21712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-1/data/info.pb
I20240501 20:00:09.810918 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 21712
I20240501 20:00:09.811034 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-1/wal/instance
I20240501 20:00:09.811257 8865 external_mini_cluster.cc:1428] Setting key dfca8363b7ccb97ae5ad00e140d34b80
I20240501 20:00:09.811866 21780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:09.815234 21780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.168.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.124@KRBTEST.COM" created.
May 01 20:00:09 dist-test-slave-tzct krb5kdc[21624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.125@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:09.827212 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.817006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.8.168.126:45759)
0501 20:00:09.817135 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:09.817138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:09.818003 (+ 865us) server_negotiation.cc:407] Connection header received
0501 20:00:09.818803 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:09.818806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:09.818856 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:09.818931 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:09.820169 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:09.820879 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:09.821545 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:09.821684 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:09.823885 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:09.823905 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:09.823907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:09.823934 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:09.825404 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:09.825862 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:09.825865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:09.825867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:09.825911 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:09.826158 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:09.826161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:09.826163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:09.826348 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:09.826437 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:09.826929 (+ 492us) server_negotiation.cc:299] Negotiation successful
0501 20:00:09.827047 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240501 20:00:09.827803 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.817356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45759 (local address 127.0.0.1:48614)
0501 20:00:09.817859 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.817895 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:09.818601 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:09.819072 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:09.819079 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:09.819417 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:09.820019 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:09.820035 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:09.820999 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:09.821002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:09.821422 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:09.821433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:09.821657 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:09.822327 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:09.822352 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:09.823729 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:09.825515 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:09.825520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:09.825531 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:09.825771 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:09.825990 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:09.825993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:09.825994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:09.826076 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:09.826455 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:09.826461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:09.826697 (+ 236us) client_negotiation.cc:769] Sending connection context
0501 20:00:09.826880 (+ 183us) client_negotiation.cc:240] Negotiation successful
0501 20:00:09.827134 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":141,"threads_started":1}
I20240501 20:00:09.829116 21780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.124.keytab.
Entry for principal kudu/127.8.168.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.124.keytab.
May 01 20:00:09 dist-test-slave-tzct krb5kdc[21624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.125@KRBTEST.COM for kudu/127.8.168.125@KRBTEST.COM
I20240501 20:00:09.834081 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.124:33909
--webserver_interface=127.8.168.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.124.keytab
--principal=kudu/127.8.168.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:45759,127.8.168.125:36605,127.8.168.124:33909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 20:00:09.837659 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.829605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36605 (local address 127.0.0.1:43522)
0501 20:00:09.829721 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.829734 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:09.829807 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:09.830425 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:09.830427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:09.830555 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:09.830721 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:09.830727 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:09.831791 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:09.831795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:09.832134 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:09.832142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:09.832268 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:09.832762 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:09.832777 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:09.833994 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:09.836146 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:09.836149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:09.836151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:09.836358 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:09.836595 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:09.836597 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:09.836599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:09.836649 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:09.837070 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:09.837073 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:09.837164 (+ 91us) client_negotiation.cc:769] Sending connection context
0501 20:00:09.837280 (+ 116us) client_negotiation.cc:240] Negotiation successful
0501 20:00:09.837419 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20240501 20:00:09.837677 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.829737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43522 (local address 127.8.168.125:36605)
0501 20:00:09.830059 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:09.830064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:09.830082 (+ 18us) server_negotiation.cc:407] Connection header received
0501 20:00:09.830122 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:09.830126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:09.830258 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:09.830412 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:09.830871 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:09.831652 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:09.832252 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:09.832424 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:09.834122 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:09.834143 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:09.834155 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:09.834181 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:09.836030 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:09.836467 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:09.836472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:09.836473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:09.836524 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:09.836760 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:09.836763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:09.836765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:09.836977 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:09.837085 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:09.837391 (+ 306us) server_negotiation.cc:299] Negotiation successful
0501 20:00:09.837520 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":126,"threads_started":1}
I20240501 20:00:09.839604 21780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:00:09.840359 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.840103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:33909 (local address 127.0.0.1:45454)
0501 20:00:09.840217 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.840288 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20240501 20:00:09.840555 21731 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
I20240501 20:00:09.841425 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.841189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:33909 (local address 127.0.0.1:45464)
0501 20:00:09.841324 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.841369 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20240501 20:00:09.841575 21780 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:33909: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
I20240501 20:00:09.852996 21705 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } attempt: 3
May 01 20:00:09 dist-test-slave-tzct krb5kdc[21624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for kudu/127.8.168.125@KRBTEST.COM
I20240501 20:00:09.861292 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.853562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36605 (local address 127.0.0.1:43538)
0501 20:00:09.853704 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.853720 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:09.853815 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:09.854101 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:09.854103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:09.854306 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:09.854527 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:09.854534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:09.855457 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:09.855459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:09.855813 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:09.855819 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:09.855909 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:09.856860 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:09.856874 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:09.858078 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:09.859924 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:09.859927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:09.859929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:09.860108 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:09.860339 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:09.860341 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:09.860343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:09.860388 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:09.860740 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:09.860743 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:09.860904 (+ 161us) client_negotiation.cc:769] Sending connection context
0501 20:00:09.861010 (+ 106us) client_negotiation.cc:240] Negotiation successful
0501 20:00:09.861130 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20240501 20:00:09.861361 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.853640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43538 (local address 127.8.168.125:36605)
0501 20:00:09.853786 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:09.853790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:09.853810 (+ 20us) server_negotiation.cc:407] Connection header received
0501 20:00:09.853906 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:09.853910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:09.853980 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:09.854073 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:09.854649 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:09.855355 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:09.855912 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:09.856110 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:09.858205 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:09.858229 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:09.858232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:09.858260 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:09.859756 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:09.860201 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:09.860204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:09.860206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:09.860256 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:09.860457 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:09.860460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:09.860462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:09.860640 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:09.860764 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:09.861075 (+ 311us) server_negotiation.cc:299] Negotiation successful
0501 20:00:09.861193 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20240501 20:00:09.862090 21705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:00:09.862839 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.862597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:33909 (local address 127.0.0.1:45470)
0501 20:00:09.862718 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.862775 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20240501 20:00:09.863457 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.863237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:33909 (local address 127.0.0.1:45472)
0501 20:00:09.863366 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.863404 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20240501 20:00:09.863560 21705 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:33909: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
I20240501 20:00:09.890818 21780 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } attempt: 1
I20240501 20:00:09.891726 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.891387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:33909 (local address 127.0.0.1:45482)
0501 20:00:09.891575 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.891638 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20240501 20:00:09.892478 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.892251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:33909 (local address 127.0.0.1:45498)
0501 20:00:09.892372 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.892416 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20240501 20:00:09.892604 21780 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:33909: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
I20240501 20:00:09.894716 21705 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } attempt: 1
I20240501 20:00:09.895423 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.895144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:33909 (local address 127.0.0.1:45512)
0501 20:00:09.895299 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.895348 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20240501 20:00:09.896229 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.895986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:33909 (local address 127.0.0.1:45524)
0501 20:00:09.896122 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.896165 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20240501 20:00:09.896358 21705 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:33909: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
W20240501 20:00:09.943583 21786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:09.943828 21786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:09.943898 21786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:09.946950 21786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:00:09.947021 21786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:09.947050 21786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:09.947072 21786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:00:09.947103 21786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:00:09.951385 21786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:45759,127.8.168.125:36605,127.8.168.124:33909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.124.keytab
--principal=kudu/127.8.168.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.124:33909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.168.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:09.952303 21786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:09.953184 21786 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:09.960578 21792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:09.960770 21780 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } attempt: 2
W20240501 20:00:09.960686 21795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:09.961051 21793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:09.961478 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.961208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:33909 (local address 127.0.0.1:45526)
0501 20:00:09.961346 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.961393 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20240501 20:00:09.961663 21786 server_base.cc:1034] running on GCE node
I20240501 20:00:09.962217 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.962021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:33909 (local address 127.0.0.1:45528)
0501 20:00:09.962118 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.962159 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20240501 20:00:09.962242 21786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20240501 20:00:09.962368 21780 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:33909: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
I20240501 20:00:09.963201 21786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:09.964340 21786 hybrid_clock.cc:648] HybridClock initialized: now 1714593609964315 us; error 45 us; skew 500 ppm
May 01 20:00:09 dist-test-slave-tzct krb5kdc[21624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:09.966809 21786 init.cc:377] Logged in from keytab as kudu/127.8.168.124@KRBTEST.COM (short username kudu)
I20240501 20:00:09.967707 21786 webserver.cc:460] Webserver started at http://127.8.168.124:38309/ using document root <none> and password file <none>
I20240501 20:00:09.968215 21786 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:09.968271 21786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:09.968446 21786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:09.970289 21786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-2/data/instance:
uuid: "119d67b6616e44ef903a1dba17f2acd2"
format_stamp: "Formatted at 2024-05-01 20:00:09 on dist-test-slave-tzct"
server_key: "8b4fe03f6b6541604aa9f737dac85950"
server_key_iv: "c52e3d1601d6b967f8b47c3d7492e70b"
server_key_version: "encryptionkey@0"
I20240501 20:00:09.970757 21786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-2/wal/instance:
uuid: "119d67b6616e44ef903a1dba17f2acd2"
format_stamp: "Formatted at 2024-05-01 20:00:09 on dist-test-slave-tzct"
server_key: "8b4fe03f6b6541604aa9f737dac85950"
server_key_iv: "c52e3d1601d6b967f8b47c3d7492e70b"
server_key_version: "encryptionkey@0"
I20240501 20:00:09.971498 21705 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } attempt: 2
I20240501 20:00:09.972190 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.971932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:33909 (local address 127.0.0.1:45530)
0501 20:00:09.972064 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.972113 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20240501 20:00:09.972896 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:09.972596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:33909 (local address 127.0.0.1:45536)
0501 20:00:09.972796 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:09.972838 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20240501 20:00:09.973011 21705 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:33909: Network error: Client connection negotiation failed: client connection to 127.8.168.124:33909: connect: Connection refused (error 111)
I20240501 20:00:09.974267 21786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 20:00:09.976549 21802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:09.977646 21786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:09.977766 21786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-2/wal
uuid: "119d67b6616e44ef903a1dba17f2acd2"
format_stamp: "Formatted at 2024-05-01 20:00:09 on dist-test-slave-tzct"
server_key: "8b4fe03f6b6541604aa9f737dac85950"
server_key_iv: "c52e3d1601d6b967f8b47c3d7492e70b"
server_key_version: "encryptionkey@0"
I20240501 20:00:09.977864 21786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:09.993453 21786 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:09.996517 21786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:09.996685 21786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:10.003209 21786 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.124:33909
I20240501 20:00:10.003221 21854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.124:33909 every 8 connection(s)
I20240501 20:00:10.004177 21786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-2/data/info.pb
I20240501 20:00:10.006980 21855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:10.010198 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 21786
I20240501 20:00:10.010342 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1714593353892612-8865-0/minicluster-data/master-2/wal/instance
I20240501 20:00:10.010315 21855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:00:10.010566 8865 external_mini_cluster.cc:1428] Setting key a165ca15414f6b4a6083dd1df0e2737a
May 01 20:00:10 dist-test-slave-tzct krb5kdc[21624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.124@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
May 01 20:00:10 dist-test-slave-tzct krb5kdc[21624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.124@KRBTEST.COM
I20240501 20:00:10.021946 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:10.012047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48622 (local address 127.8.168.126:45759)
0501 20:00:10.012165 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:10.012169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:10.012401 (+ 232us) server_negotiation.cc:407] Connection header received
0501 20:00:10.013012 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:10.013015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:10.013064 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:10.013155 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:10.014355 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.015066 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:10.015755 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.015889 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:10.018412 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:10.018436 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:10.018440 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:10.018471 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:10.019930 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.020416 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.020419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.020421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.020467 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.020790 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.020794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.020796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.020983 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:10.021075 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:10.021621 (+ 546us) server_negotiation.cc:299] Negotiation successful
0501 20:00:10.021750 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240501 20:00:10.022696 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:10.011936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45759 (local address 127.0.0.1:48622)
0501 20:00:10.012272 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:10.012310 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:10.012831 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:10.013243 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:10.013249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:10.013745 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:10.014180 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:10.014197 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.015194 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:10.015196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:10.015645 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:10.015655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.015876 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:10.016578 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:10.016614 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:10.018266 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:10.020040 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:10.020047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:10.020055 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:10.020314 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:10.020543 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:10.020546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:10.020548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:10.020652 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:10.021080 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:10.021086 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:10.021387 (+ 301us) client_negotiation.cc:769] Sending connection context
0501 20:00:10.021566 (+ 179us) client_negotiation.cc:240] Negotiation successful
0501 20:00:10.021835 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":113,"threads_started":1}
I20240501 20:00:10.022697 21858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:10.011684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45546 (local address 127.8.168.124:33909)
0501 20:00:10.012120 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:10.012130 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:10.012166 (+ 36us) server_negotiation.cc:407] Connection header received
0501 20:00:10.012825 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:10.012845 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:10.013024 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:10.013200 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:10.014282 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.015327 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:10.016137 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.016341 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:10.018789 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:10.018821 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:10.018833 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:10.018858 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:10.020510 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.020927 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.020931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.020932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.020975 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.021178 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.021181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.021182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.021399 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:10.021572 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:10.021778 (+ 206us) server_negotiation.cc:299] Negotiation successful
0501 20:00:10.021898 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":127,"threads_started":1}
I20240501 20:00:10.023850 21855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } has no permanent_uuid. Determining permanent_uuid...
May 01 20:00:10 dist-test-slave-tzct krb5kdc[21624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.124@KRBTEST.COM for kudu/127.8.168.125@KRBTEST.COM
May 01 20:00:10 dist-test-slave-tzct krb5kdc[21624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:10.031538 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:10.024467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.8.168.125:36605)
0501 20:00:10.024578 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:10.024581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:10.024593 (+ 12us) server_negotiation.cc:407] Connection header received
0501 20:00:10.024630 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:10.024633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:10.024670 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:10.024759 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:10.025343 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.025802 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:10.026345 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.026529 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:10.028348 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:10.028366 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:10.028369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:10.028393 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:10.029971 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.030468 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.030473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.030475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.030529 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.030753 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.030757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.030759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.030915 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:10.031011 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:10.031254 (+ 243us) server_negotiation.cc:299] Negotiation successful
0501 20:00:10.031372 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240501 20:00:10.031538 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:10.024365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36605 (local address 127.0.0.1:43544)
0501 20:00:10.024494 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:10.024507 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:10.024572 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:10.024818 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:10.024821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:10.024972 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:10.025181 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:10.025190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.025908 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:10.025911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:10.026244 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:10.026251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.026352 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:10.026901 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:10.026913 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:10.028232 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:10.030072 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:10.030076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:10.030078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:10.030347 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:10.030618 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:10.030621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:10.030623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:10.030671 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:10.031011 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:10.031014 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:10.031095 (+ 81us) client_negotiation.cc:769] Sending connection context
0501 20:00:10.031200 (+ 105us) client_negotiation.cc:240] Negotiation successful
0501 20:00:10.031328 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20240501 20:00:10.032670 21855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:00:10.033104 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:10.025267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48636 (local address 127.8.168.126:45759)
0501 20:00:10.025373 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:10.025376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:10.025423 (+ 47us) server_negotiation.cc:407] Connection header received
0501 20:00:10.025525 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:10.025528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:10.025662 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:10.025752 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:10.026440 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.027028 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:10.027710 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.027863 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:10.029906 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:10.029922 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:10.029924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:10.029947 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:10.031416 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.031802 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.031806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.031808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.031874 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.032103 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.032105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.032107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.032271 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:10.032473 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:10.032839 (+ 366us) server_negotiation.cc:299] Negotiation successful
0501 20:00:10.032954 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
May 01 20:00:10 dist-test-slave-tzct krb5kdc[21624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.124@KRBTEST.COM for kudu/127.8.168.124@KRBTEST.COM
May 01 20:00:10 dist-test-slave-tzct krb5kdc[21624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.125@KRBTEST.COM
I20240501 20:00:10.039981 21858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:10.033416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.8.168.124:33909)
0501 20:00:10.033528 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:10.033532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:10.033578 (+ 46us) server_negotiation.cc:407] Connection header received
0501 20:00:10.033705 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:10.033708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:10.033752 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:10.033838 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:10.034377 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.034839 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:10.035414 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.035592 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:10.037229 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:10.037248 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:10.037251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:10.037276 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:10.038570 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.038928 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.038930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.038931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.038972 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.039197 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.039200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.039202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.039389 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:10.039473 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:10.039732 (+ 259us) server_negotiation.cc:299] Negotiation successful
0501 20:00:10.039836 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240501 20:00:10.040004 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:10.033347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:33909 (local address 127.0.0.1:45552)
0501 20:00:10.033525 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:10.033539 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:10.033617 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:10.033885 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:10.033888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:10.034055 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:10.034268 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:10.034275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.034932 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:10.034935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:10.035316 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:10.035323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.035422 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:10.035910 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:10.035922 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:10.037120 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:10.038668 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:10.038670 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:10.038672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:10.038835 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:10.039049 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:10.039052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:10.039054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:10.039110 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:10.039480 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:10.039484 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:10.039572 (+ 88us) client_negotiation.cc:769] Sending connection context
0501 20:00:10.039681 (+ 109us) client_negotiation.cc:240] Negotiation successful
0501 20:00:10.039791 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20240501 20:00:10.041414 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:10.034222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43548 (local address 127.8.168.125:36605)
0501 20:00:10.034322 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:10.034326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:10.034338 (+ 12us) server_negotiation.cc:407] Connection header received
0501 20:00:10.034439 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:10.034442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:10.034482 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:10.034543 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:10.035158 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.035574 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:10.036137 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.036339 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:10.038436 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:10.038453 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:10.038455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:10.038477 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:10.040036 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.040405 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.040409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.040411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.040457 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.040672 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.040675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.040676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.040884 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:10.041088 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:10.041151 (+ 63us) server_negotiation.cc:299] Negotiation successful
0501 20:00:10.041261 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240501 20:00:10.043479 21855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2: Bootstrap starting.
I20240501 20:00:10.044600 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:00:10.046006 21855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:10.047124 21855 log.cc:824] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:10.049587 21855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2: No bootstrap required, opened a new log
I20240501 20:00:10.053511 21780 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } attempt: 3
I20240501 20:00:10.053452 21855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } }
I20240501 20:00:10.053642 21855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:10.053699 21855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119d67b6616e44ef903a1dba17f2acd2, State: Initialized, Role: FOLLOWER
I20240501 20:00:10.054498 21855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } }
I20240501 20:00:10.055182 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:10.047038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43558 (local address 127.8.168.125:36605)
0501 20:00:10.047164 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:10.047169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:10.047357 (+ 188us) server_negotiation.cc:407] Connection header received
0501 20:00:10.047400 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:10.047403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:10.047453 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:10.047524 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:10.048611 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.049349 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:10.050351 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.050530 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:10.051623 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:10.051644 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:10.051648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:10.051677 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:10.053237 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.053641 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.053645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.053647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.053691 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.053970 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.053973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.053975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.054172 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:10.054296 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:10.054896 (+ 600us) server_negotiation.cc:299] Negotiation successful
0501 20:00:10.055010 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 20:00:10.056114 21855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:00:10.056664 21858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:10.047411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45562 (local address 127.8.168.124:33909)
0501 20:00:10.047532 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:10.047536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:10.047769 (+ 233us) server_negotiation.cc:407] Connection header received
0501 20:00:10.047933 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:10.047936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:10.047984 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:10.048035 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:10.048992 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.049667 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:10.050727 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.050913 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:10.053578 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:10.053606 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:10.053609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:10.053637 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:10.055236 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.055656 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.055659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.055661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.055714 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.055927 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.055930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.055932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.056084 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:10.056214 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:10.056417 (+ 203us) server_negotiation.cc:299] Negotiation successful
0501 20:00:10.056519 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20240501 20:00:10.057888 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:10.047128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48648 (local address 127.8.168.126:45759)
0501 20:00:10.047338 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:10.047350 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:10.047794 (+ 444us) server_negotiation.cc:407] Connection header received
0501 20:00:10.048016 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:10.048019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:10.048075 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:10.048135 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:10.049304 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.049942 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:10.050799 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.050989 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:10.052468 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:10.052490 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:10.052493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:10.052524 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:10.055209 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.056654 (+ 1445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.056658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.056659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.056715 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.057008 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.057011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.057013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.057193 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:10.057300 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:10.057585 (+ 285us) server_negotiation.cc:299] Negotiation successful
0501 20:00:10.057709 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20240501 20:00:10.058784 21869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } } }
May 01 20:00:10 dist-test-slave-tzct krb5kdc[21624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.125@KRBTEST.COM for kudu/127.8.168.124@KRBTEST.COM
I20240501 20:00:10.060526 21869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [sys.catalog]: This master's current role is: FOLLOWER
W20240501 20:00:10.061435 21880 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:00:10.061507 21880 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:00:10.060854 21786 master_runner.cc:186] Error getting master registration for 127.8.168.126:45759: OK, instance_id { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" instance_seqno: 1714593609612522 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240501 20:00:10.063517 21786 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:00:10.065479 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:10.053901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:33909 (local address 127.0.0.1:45574)
0501 20:00:10.054187 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:10.054200 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:10.054275 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:10.056871 (+ 2596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:10.056875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:10.057047 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:10.057269 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:10.057278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.058366 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:10.058370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:10.058874 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:10.058885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.058971 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:10.059464 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:10.059479 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:10.060997 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:10.063655 (+ 2658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:10.063659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:10.063662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:10.063869 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:10.064562 (+ 693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:10.064565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:10.064567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:10.064631 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:10.065060 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:10.065063 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:10.065145 (+ 82us) client_negotiation.cc:769] Sending connection context
0501 20:00:10.065255 (+ 110us) client_negotiation.cc:240] Negotiation successful
0501 20:00:10.065358 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20240501 20:00:10.065610 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:10.054396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.8.168.124:33909)
0501 20:00:10.056592 (+ 2196us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:10.056596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:10.056609 (+ 13us) server_negotiation.cc:407] Connection header received
0501 20:00:10.056642 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:10.056645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:10.056688 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:10.057383 (+ 695us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:10.057473 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.058240 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:10.059172 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.059320 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:10.061129 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:10.061147 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:10.061150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:10.061175 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:10.063482 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.063981 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.063985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.063987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.064040 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.064769 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.064773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.064775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.064963 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:10.065067 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:10.065312 (+ 245us) server_negotiation.cc:299] Negotiation successful
0501 20:00:10.065433 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2118,"thread_start_us":115,"threads_started":1}
I20240501 20:00:10.067948 21780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1: Bootstrap starting.
I20240501 20:00:10.070058 21780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:10.070736 21780 log.cc:824] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:10.072156 21705 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } attempt: 3
I20240501 20:00:10.073355 21780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1: No bootstrap required, opened a new log
I20240501 20:00:10.076318 21780 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } }
I20240501 20:00:10.076467 21780 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:10.076560 21780 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9567ba68c2a4af39b43b0f79f3938f1, State: Initialized, Role: FOLLOWER
I20240501 20:00:10.076990 21780 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } }
May 01 20:00:10 dist-test-slave-tzct krb5kdc[21624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for kudu/127.8.168.124@KRBTEST.COM
I20240501 20:00:10.078116 21883 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } } }
I20240501 20:00:10.078274 21883 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:00:10.078856 21780 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:00:10.082247 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:10.072607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.8.168.124:33909)
0501 20:00:10.072997 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:10.073001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:10.073014 (+ 13us) server_negotiation.cc:407] Connection header received
0501 20:00:10.073046 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:10.073049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:10.073091 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:10.073157 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:10.073831 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.074514 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:10.075409 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.076001 (+ 592us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:10.078370 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:10.078388 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:10.078391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:10.078417 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:10.080179 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.080636 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.080640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.080642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.080689 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:10.080985 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:10.080989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:10.080991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:10.081194 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:10.081279 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:10.081972 (+ 693us) server_negotiation.cc:299] Negotiation successful
0501 20:00:10.082084 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311}
I20240501 20:00:10.082787 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:10.072551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:33909 (local address 127.0.0.1:45584)
0501 20:00:10.072758 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:10.072773 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:10.072849 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:10.073285 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:10.073289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:10.073476 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:10.073707 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:10.073716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.074665 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:10.074669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:10.075271 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:10.075282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:10.075413 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:10.075997 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:10.076014 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:10.078238 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:10.080313 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:10.080317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:10.080320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:10.080534 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:10.080830 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:10.080833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:10.080836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:10.080894 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:10.081322 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:10.081325 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:10.081414 (+ 89us) client_negotiation.cc:769] Sending connection context
0501 20:00:10.081675 (+ 261us) client_negotiation.cc:240] Negotiation successful
0501 20:00:10.082615 (+ 940us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
W20240501 20:00:10.083765 21894 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:00:10.083880 21894 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:00:10.081998 21712 master_runner.cc:186] Error getting master registration for 127.8.168.126:45759: OK, instance_id { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" instance_seqno: 1714593609612522 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240501 20:00:10.084950 21712 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:00:10.085960 21705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b: Bootstrap starting.
I20240501 20:00:10.088466 21705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:10.090837 21705 log.cc:824] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:10.092952 21705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b: No bootstrap required, opened a new log
I20240501 20:00:10.097347 21705 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } }
I20240501 20:00:10.097622 21705 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:10.097829 21705 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7df2c6884a543ca84a325c08a2bc11b, State: Initialized, Role: FOLLOWER
I20240501 20:00:10.098528 21705 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } }
I20240501 20:00:10.099129 21897 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } } }
I20240501 20:00:10.099313 21897 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:00:10.100101 21705 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b [sys.catalog]: configured and running, proceeding with master startup.
W20240501 20:00:10.105149 21908 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:00:10.105227 21908 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 20:00:10.109362 21658 tablet.cc:2369] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:00:10.110272 21637 master_runner.cc:263] No leader master found from master c7df2c6884a543ca84a325c08a2bc11b
I20240501 20:00:10.110342 21637 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240501 20:00:10.247988 21808 tablet.cc:2369] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:00:10.257038 21869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:10.257264 21869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } }
I20240501 20:00:10.258236 21869 leader_election.cc:288] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7df2c6884a543ca84a325c08a2bc11b (127.8.168.126:45759), f9567ba68c2a4af39b43b0f79f3938f1 (127.8.168.125:36605)
I20240501 20:00:10.259689 21679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "119d67b6616e44ef903a1dba17f2acd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7df2c6884a543ca84a325c08a2bc11b" is_pre_election: true
I20240501 20:00:10.260294 21679 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 119d67b6616e44ef903a1dba17f2acd2 in term 0.
I20240501 20:00:10.260949 21804 leader_election.cc:302] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 119d67b6616e44ef903a1dba17f2acd2, c7df2c6884a543ca84a325c08a2bc11b; no voters:
I20240501 20:00:10.261484 21869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:10.261585 21869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:10.261734 21869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:10.262104 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "119d67b6616e44ef903a1dba17f2acd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" is_pre_election: true
I20240501 20:00:10.262392 21754 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 119d67b6616e44ef903a1dba17f2acd2 in term 0.
I20240501 20:00:10.263059 21869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } }
I20240501 20:00:10.263378 21869 leader_election.cc:288] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [CANDIDATE]: Term 1 election: Requested vote from peers c7df2c6884a543ca84a325c08a2bc11b (127.8.168.126:45759), f9567ba68c2a4af39b43b0f79f3938f1 (127.8.168.125:36605)
I20240501 20:00:10.264036 21679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "119d67b6616e44ef903a1dba17f2acd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7df2c6884a543ca84a325c08a2bc11b"
I20240501 20:00:10.264101 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "119d67b6616e44ef903a1dba17f2acd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9567ba68c2a4af39b43b0f79f3938f1"
I20240501 20:00:10.264160 21679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:10.264201 21754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:10.265375 21679 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 119d67b6616e44ef903a1dba17f2acd2 in term 1.
I20240501 20:00:10.265375 21754 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 119d67b6616e44ef903a1dba17f2acd2 in term 1.
I20240501 20:00:10.265834 21803 leader_election.cc:302] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 119d67b6616e44ef903a1dba17f2acd2, f9567ba68c2a4af39b43b0f79f3938f1; no voters:
I20240501 20:00:10.266705 21869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:10.266918 21869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [term 1 LEADER]: Becoming Leader. State: Replica: 119d67b6616e44ef903a1dba17f2acd2, State: Running, Role: LEADER
I20240501 20:00:10.267235 21869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } }
I20240501 20:00:10.269220 21910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 119d67b6616e44ef903a1dba17f2acd2. Latest consensus state: current_term: 1 leader_uuid: "119d67b6616e44ef903a1dba17f2acd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } } }
I20240501 20:00:10.269404 21910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:10.269794 21912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:00:10.271057 21912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:00:10.275314 21679 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b [term 1 FOLLOWER]: Refusing update from remote peer 119d67b6616e44ef903a1dba17f2acd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:00:10.275314 21754 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1 [term 1 FOLLOWER]: Refusing update from remote peer 119d67b6616e44ef903a1dba17f2acd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:00:10.275987 21869 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:10.276193 21910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:10.280006 21897 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 119d67b6616e44ef903a1dba17f2acd2. Latest consensus state: current_term: 1 leader_uuid: "119d67b6616e44ef903a1dba17f2acd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } } }
I20240501 20:00:10.280017 21883 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 119d67b6616e44ef903a1dba17f2acd2. Latest consensus state: current_term: 1 leader_uuid: "119d67b6616e44ef903a1dba17f2acd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } } }
I20240501 20:00:10.280146 21897 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:00:10.280166 21883 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:00:10.283571 21910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "119d67b6616e44ef903a1dba17f2acd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } } }
I20240501 20:00:10.283703 21910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:10.284142 21869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "119d67b6616e44ef903a1dba17f2acd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } } }
I20240501 20:00:10.284236 21869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:10.285287 21897 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "119d67b6616e44ef903a1dba17f2acd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } } }
I20240501 20:00:10.285388 21897 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:00:10.285478 21914 mvcc.cc:205] Tried to move back new op lower bound from 7022975427681456128 to 7022975427656900608. Current Snapshot: MvccSnapshot[applied={T|T < 7022975427681456128}]
I20240501 20:00:10.285665 21912 catalog_manager.cc:1347] Generated new cluster ID: b3ce6e9205174d4fb5e7772e6d809650
I20240501 20:00:10.285717 21912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:00:10.286800 21883 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "119d67b6616e44ef903a1dba17f2acd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7df2c6884a543ca84a325c08a2bc11b" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45759 } } peers { permanent_uuid: "f9567ba68c2a4af39b43b0f79f3938f1" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36605 } } peers { permanent_uuid: "119d67b6616e44ef903a1dba17f2acd2" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 33909 } } }
I20240501 20:00:10.286916 21883 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:00:10.287331 21913 mvcc.cc:205] Tried to move back new op lower bound from 7022975427681456128 to 7022975427656900608. Current Snapshot: MvccSnapshot[applied={T|T < 7022975427681456128 or (T in {7022975427681456128})}]
I20240501 20:00:10.287520 21915 mvcc.cc:205] Tried to move back new op lower bound from 7022975427681456128 to 7022975427656900608. Current Snapshot: MvccSnapshot[applied={T|T < 7022975427681456128}]
I20240501 20:00:10.303333 21912 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:00:10.304589 21912 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:00:10.313200 21912 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 119d67b6616e44ef903a1dba17f2acd2: Generated new TSK 0
I20240501 20:00:10.313896 21912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:00:11.085630 21894 catalog_manager.cc:1259] Loaded cluster ID: b3ce6e9205174d4fb5e7772e6d809650
I20240501 20:00:11.085708 21894 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1: loading cluster ID for follower catalog manager: success
I20240501 20:00:11.087483 21894 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1: acquiring CA information for follower catalog manager: success
I20240501 20:00:11.088670 21894 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f9567ba68c2a4af39b43b0f79f3938f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 20:00:11.106640 21908 catalog_manager.cc:1259] Loaded cluster ID: b3ce6e9205174d4fb5e7772e6d809650
I20240501 20:00:11.106739 21908 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b: loading cluster ID for follower catalog manager: success
I20240501 20:00:11.109130 21908 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b: acquiring CA information for follower catalog manager: success
I20240501 20:00:11.110365 21908 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7df2c6884a543ca84a325c08a2bc11b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 20:00:11.382547 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 21637
I20240501 20:00:11.388015 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 21712
I20240501 20:00:11.393020 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 21786
2024-05-01T20:00:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2942 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:00:11 dist-test-slave-tzct krb5kdc[21926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:00:11 dist-test-slave-tzct krb5kdc[21926](info): set up 2 sockets
May 01 20:00:11 dist-test-slave-tzct krb5kdc[21926](info): commencing operation
krb5kdc: starting...
W20240501 20:00:12.288962 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.876s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:00:12 dist-test-slave-tzct krb5kdc[21926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:00:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:00:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 20:00:12.407653 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:34153
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:34153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:12.514827 21939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:12.515084 21939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:12.515141 21939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:12.518231 21939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:00:12.518299 21939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:12.518321 21939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:12.518338 21939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:00:12.518354 21939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:00:12.522783 21939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:34153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:34153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:12.523769 21939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:12.524691 21939 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:12.532305 21944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:12.532742 21947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:12.532843 21945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:12.533524 21939 server_base.cc:1034] running on GCE node
I20240501 20:00:12.534116 21939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:12.535082 21939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:12.536280 21939 hybrid_clock.cc:648] HybridClock initialized: now 1714593612536258 us; error 37 us; skew 500 ppm
May 01 20:00:12 dist-test-slave-tzct krb5kdc[21926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593612, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:12.539078 21939 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:00:12.540105 21939 webserver.cc:460] Webserver started at http://127.8.168.126:33855/ using document root <none> and password file <none>
I20240501 20:00:12.540642 21939 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:12.540705 21939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:12.540922 21939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:12.542824 21939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "10724f160e1347359856743a3a19d9f9"
format_stamp: "Formatted at 2024-05-01 20:00:12 on dist-test-slave-tzct"
server_key: "de278c2a939e0a268a9d2c5a5a722d33"
server_key_iv: "04eabed30567fd7a244d6f2c7a2d8014"
server_key_version: "encryptionkey@0"
I20240501 20:00:12.543349 21939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "10724f160e1347359856743a3a19d9f9"
format_stamp: "Formatted at 2024-05-01 20:00:12 on dist-test-slave-tzct"
server_key: "de278c2a939e0a268a9d2c5a5a722d33"
server_key_iv: "04eabed30567fd7a244d6f2c7a2d8014"
server_key_version: "encryptionkey@0"
I20240501 20:00:12.546620 21939 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240501 20:00:12.548897 21954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:12.550104 21939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:00:12.550207 21939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "10724f160e1347359856743a3a19d9f9"
format_stamp: "Formatted at 2024-05-01 20:00:12 on dist-test-slave-tzct"
server_key: "de278c2a939e0a268a9d2c5a5a722d33"
server_key_iv: "04eabed30567fd7a244d6f2c7a2d8014"
server_key_version: "encryptionkey@0"
I20240501 20:00:12.550324 21939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:12.561291 21939 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:12.563901 21939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:12.564043 21939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:12.570304 21939 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:34153
I20240501 20:00:12.570330 22006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:34153 every 8 connection(s)
I20240501 20:00:12.571249 21939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:00:12.573650 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 21939
I20240501 20:00:12.573765 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:00:12.574100 8865 external_mini_cluster.cc:1428] Setting key f40da600b9b4200ca0b7067070580719
I20240501 20:00:12.574170 22007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:12.579952 22007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9: Bootstrap starting.
May 01 20:00:12 dist-test-slave-tzct krb5kdc[21926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:12.582157 22007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:12.582918 22007 log.cc:824] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:12.584802 22007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9: No bootstrap required, opened a new log
I20240501 20:00:12.586736 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:12.575375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.8.168.126:34153)
0501 20:00:12.575793 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:12.575802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:12.575836 (+ 34us) server_negotiation.cc:407] Connection header received
0501 20:00:12.576677 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:12.576694 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:12.577000 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:12.577324 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:12.578287 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:12.579018 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:12.579649 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:12.579911 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:12.582220 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:12.582237 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:12.582249 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:12.582294 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:12.584302 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:12.584759 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:12.584766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:12.584771 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:12.584850 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:12.585106 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:12.585108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:12.585109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:12.585468 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:12.585642 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:12.585888 (+ 246us) server_negotiation.cc:299] Negotiation successful
0501 20:00:12.586129 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":155,"threads_started":1}
I20240501 20:00:12.588141 22007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10724f160e1347359856743a3a19d9f9" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 34153 } }
I20240501 20:00:12.588315 22007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:12.588387 22007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10724f160e1347359856743a3a19d9f9, State: Initialized, Role: FOLLOWER
I20240501 20:00:12.588783 22007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10724f160e1347359856743a3a19d9f9" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 34153 } }
I20240501 20:00:12.588927 22007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:00:12.588989 22007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:00:12.589099 22007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:12.589745 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:00:12.589988 22007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10724f160e1347359856743a3a19d9f9" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 34153 } }
I20240501 20:00:12.590420 22007 leader_election.cc:302] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10724f160e1347359856743a3a19d9f9; no voters:
I20240501 20:00:12.590770 22007 leader_election.cc:288] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:00:12.591135 22013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:12.591384 22013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 10724f160e1347359856743a3a19d9f9, State: Running, Role: LEADER
I20240501 20:00:12.591696 22013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10724f160e1347359856743a3a19d9f9" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 34153 } }
I20240501 20:00:12.591847 22007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:00:12.593088 22014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10724f160e1347359856743a3a19d9f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10724f160e1347359856743a3a19d9f9" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 34153 } } }
I20240501 20:00:12.593205 22014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:12.593567 22021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:00:12.593590 22012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10724f160e1347359856743a3a19d9f9. Latest consensus state: current_term: 1 leader_uuid: "10724f160e1347359856743a3a19d9f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10724f160e1347359856743a3a19d9f9" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 34153 } } }
I20240501 20:00:12.593773 22012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:12.596186 22021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:00:12.600679 22021 catalog_manager.cc:1347] Generated new cluster ID: e49a6f16896d4e8b90dd1364fb8814f6
I20240501 20:00:12.600776 22021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:00:12.631477 22021 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:00:12.632530 22021 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:00:12.642814 22021 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 10724f160e1347359856743a3a19d9f9: Generated new TSK 0
I20240501 20:00:12.643596 22021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:00:12.696403 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 21939
2024-05-01T20:00:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1302 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:00:12 dist-test-slave-tzct krb5kdc[22034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:00:12 dist-test-slave-tzct krb5kdc[22034](info): set up 2 sockets
May 01 20:00:12 dist-test-slave-tzct krb5kdc[22034](info): commencing operation
krb5kdc: starting...
W20240501 20:00:13.584957 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.870s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:00:13 dist-test-slave-tzct krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:00:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:00:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 20:00:13.694022 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:41909
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:41909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:13.799213 22047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:13.799480 22047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:13.799561 22047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:13.802680 22047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:00:13.802749 22047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:13.802779 22047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:13.802801 22047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:00:13.802825 22047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:00:13.806962 22047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:41909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:41909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:13.807870 22047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:13.808703 22047 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:13.815483 22055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:13.815927 22053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:13.816556 22052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:13.817098 22047 server_base.cc:1034] running on GCE node
I20240501 20:00:13.817571 22047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:13.818470 22047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:13.819614 22047 hybrid_clock.cc:648] HybridClock initialized: now 1714593613819599 us; error 36 us; skew 500 ppm
May 01 20:00:13 dist-test-slave-tzct krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593613, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:13.822208 22047 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:00:13.823100 22047 webserver.cc:460] Webserver started at http://127.8.168.126:39257/ using document root <none> and password file <none>
I20240501 20:00:13.823606 22047 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:13.823660 22047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:13.823823 22047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:13.825604 22047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "105abcddc47f43b3a2dcc204988e5c12"
format_stamp: "Formatted at 2024-05-01 20:00:13 on dist-test-slave-tzct"
server_key: "a5ed3b11e9c81c1afb542785a1e3c684"
server_key_iv: "67ef9404aad49452738f6c629dd3a27e"
server_key_version: "encryptionkey@0"
I20240501 20:00:13.826047 22047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "105abcddc47f43b3a2dcc204988e5c12"
format_stamp: "Formatted at 2024-05-01 20:00:13 on dist-test-slave-tzct"
server_key: "a5ed3b11e9c81c1afb542785a1e3c684"
server_key_iv: "67ef9404aad49452738f6c629dd3a27e"
server_key_version: "encryptionkey@0"
I20240501 20:00:13.829255 22047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:00:13.831553 22062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:13.832561 22047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:13.832667 22047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "105abcddc47f43b3a2dcc204988e5c12"
format_stamp: "Formatted at 2024-05-01 20:00:13 on dist-test-slave-tzct"
server_key: "a5ed3b11e9c81c1afb542785a1e3c684"
server_key_iv: "67ef9404aad49452738f6c629dd3a27e"
server_key_version: "encryptionkey@0"
I20240501 20:00:13.832798 22047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:13.845458 22047 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:13.848039 22047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:13.848199 22047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:13.854705 22047 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:41909
I20240501 20:00:13.854754 22114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:41909 every 8 connection(s)
I20240501 20:00:13.855968 22047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:00:13.858475 22115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:13.859483 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 22047
I20240501 20:00:13.859608 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:00:13.859879 8865 external_mini_cluster.cc:1428] Setting key 8fc7113bc3e23630d17e0daf8bc9ecae
I20240501 20:00:13.863976 22115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12: Bootstrap starting.
I20240501 20:00:13.866320 22115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:13.867089 22115 log.cc:824] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12: Log is configured to *not* fsync() on all Append() calls
May 01 20:00:13 dist-test-slave-tzct krb5kdc[22034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:13.869118 22115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12: No bootstrap required, opened a new log
I20240501 20:00:13.872319 22118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:13.861068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42316 (local address 127.8.168.126:41909)
0501 20:00:13.861419 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:13.861428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:13.861462 (+ 34us) server_negotiation.cc:407] Connection header received
0501 20:00:13.861993 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:13.862009 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:13.862273 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:13.862549 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:13.863575 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:13.864380 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:13.865218 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:13.865405 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:13.867796 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:13.867817 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:13.867831 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:13.867877 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:13.869851 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:13.870208 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:13.870214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:13.870219 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:13.870316 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:13.870504 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:13.870507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:13.870509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:13.870965 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:13.871164 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:13.871324 (+ 160us) server_negotiation.cc:299] Negotiation successful
0501 20:00:13.871579 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":98,"threads_started":1}
I20240501 20:00:13.872927 22115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } }
I20240501 20:00:13.873132 22115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:13.873190 22115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105abcddc47f43b3a2dcc204988e5c12, State: Initialized, Role: FOLLOWER
I20240501 20:00:13.873593 22115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } }
I20240501 20:00:13.873721 22115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:00:13.873788 22115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:00:13.873904 22115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:13.874814 22115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } }
I20240501 20:00:13.875118 22115 leader_election.cc:302] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 105abcddc47f43b3a2dcc204988e5c12; no voters:
I20240501 20:00:13.875362 22115 leader_election.cc:288] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:00:13.875473 22120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:13.875659 22120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 1 LEADER]: Becoming Leader. State: Replica: 105abcddc47f43b3a2dcc204988e5c12, State: Running, Role: LEADER
I20240501 20:00:13.875970 22120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } }
I20240501 20:00:13.876428 22115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:00:13.877509 22122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 105abcddc47f43b3a2dcc204988e5c12. Latest consensus state: current_term: 1 leader_uuid: "105abcddc47f43b3a2dcc204988e5c12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } } }
I20240501 20:00:13.877506 22121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "105abcddc47f43b3a2dcc204988e5c12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } } }
I20240501 20:00:13.877676 22122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:13.877688 22121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:13.878016 22129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:00:13.880831 22129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:00:13.886014 22129 catalog_manager.cc:1347] Generated new cluster ID: 383f7000892044a9ae7c31e097a2321d
I20240501 20:00:13.886070 22129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:00:13.905280 22129 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:00:13.906306 22129 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:00:13.911082 22129 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12: Generated new TSK 0
I20240501 20:00:13.911703 22129 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
I20240501 20:00:13.939699 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:41909
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:14.042944 22141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:14.043175 22141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:14.043231 22141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:14.046315 22141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:14.046386 22141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:14.046464 22141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 20:00:14.050508 22141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:14.051362 22141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:14.052145 22141 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:14.059688 22147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:14.060196 22146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:14.060633 22149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:14.061342 22141 server_base.cc:1034] running on GCE node
I20240501 20:00:14.061692 22141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:14.062222 22141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:14.063367 22141 hybrid_clock.cc:648] HybridClock initialized: now 1714593614063346 us; error 34 us; skew 500 ppm
May 01 20:00:14 dist-test-slave-tzct krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593614, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:14.065990 22141 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 20:00:14.066875 22141 webserver.cc:460] Webserver started at http://127.8.168.65:43869/ using document root <none> and password file <none>
I20240501 20:00:14.067392 22141 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:14.067451 22141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:14.067646 22141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:14.069444 22141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "4ee8e00be6684bce8007800fcb7e9df5"
format_stamp: "Formatted at 2024-05-01 20:00:14 on dist-test-slave-tzct"
server_key: "584ed12a423040d313e424f0aac5be66"
server_key_iv: "142efdf649458d55ac5e978078b3f91d"
server_key_version: "encryptionkey@0"
I20240501 20:00:14.069911 22141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "4ee8e00be6684bce8007800fcb7e9df5"
format_stamp: "Formatted at 2024-05-01 20:00:14 on dist-test-slave-tzct"
server_key: "584ed12a423040d313e424f0aac5be66"
server_key_iv: "142efdf649458d55ac5e978078b3f91d"
server_key_version: "encryptionkey@0"
I20240501 20:00:14.073099 22141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:00:14.075237 22156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:14.076229 22141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:14.076339 22141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "4ee8e00be6684bce8007800fcb7e9df5"
format_stamp: "Formatted at 2024-05-01 20:00:14 on dist-test-slave-tzct"
server_key: "584ed12a423040d313e424f0aac5be66"
server_key_iv: "142efdf649458d55ac5e978078b3f91d"
server_key_version: "encryptionkey@0"
I20240501 20:00:14.076426 22141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:14.092234 22141 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:14.094795 22141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:14.094955 22141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:14.095469 22141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:14.096225 22141 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:14.096272 22141 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:14.096324 22141 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:14.096352 22141 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:14.104825 22141 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:44835
I20240501 20:00:14.104848 22269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:44835 every 8 connection(s)
I20240501 20:00:14.105762 22141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
May 01 20:00:14 dist-test-slave-tzct krb5kdc[22034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593614, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:14.115135 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 22141
I20240501 20:00:14.115260 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 20:00:14.115506 8865 external_mini_cluster.cc:1428] Setting key 7264fb00681a6af939ce0eda80ef944c
I20240501 20:00:14.118119 22118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.107438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:50243 (local address 127.8.168.126:41909)
0501 20:00:14.107567 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:14.107571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:14.108243 (+ 672us) server_negotiation.cc:407] Connection header received
0501 20:00:14.109527 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:14.109531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:14.109600 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:14.109676 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:14.110965 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.111479 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.112110 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.112296 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.114606 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:14.114629 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:14.114631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:14.114658 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:14.116128 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:14.116687 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:14.116691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:14.116693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:14.116768 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:14.117070 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:14.117073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:14.117075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:14.117255 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:14.117348 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:14.117823 (+ 475us) server_negotiation.cc:299] Negotiation successful
0501 20:00:14.117961 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240501 20:00:14.118669 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.107679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:50243)
0501 20:00:14.108106 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:14.108143 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:14.109296 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:14.109854 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:14.109862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:14.110249 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:14.110814 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:14.110828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.111596 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.111599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:14.111996 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:14.112003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.112219 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.112877 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:14.112914 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:14.114430 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:14.116240 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:14.116245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:14.116256 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:14.116579 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:14.116849 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:14.116852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:14.116854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:14.116980 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:14.117346 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:14.117350 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:14.117594 (+ 244us) client_negotiation.cc:769] Sending connection context
0501 20:00:14.117759 (+ 165us) client_negotiation.cc:240] Negotiation successful
0501 20:00:14.117977 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":113,"threads_started":1}
I20240501 20:00:14.119758 22270 heartbeater.cc:347] Connected to a master server at 127.8.168.126:41909
I20240501 20:00:14.119980 22270 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:14.120488 22270 heartbeater.cc:511] Master 127.8.168.126:41909 requested a full tablet report, sending...
I20240501 20:00:14.121915 22079 ts_manager.cc:194] Registered new tserver with Master: 4ee8e00be6684bce8007800fcb7e9df5 (127.8.168.65:44835)
I20240501 20:00:14.123024 22079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:50243
WARNING: no policy specified for kudu/127.8.168.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
I20240501 20:00:14.140513 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:41909
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:14.247488 22275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:14.247741 22275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:14.247817 22275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:14.250977 22275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:14.251051 22275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:14.251148 22275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 20:00:14.255259 22275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:14.256160 22275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:14.257035 22275 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:14.264714 22283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:14.265197 22280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:14.265794 22281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:14.266323 22275 server_base.cc:1034] running on GCE node
I20240501 20:00:14.266695 22275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:14.267266 22275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:14.268430 22275 hybrid_clock.cc:648] HybridClock initialized: now 1714593614268405 us; error 35 us; skew 500 ppm
May 01 20:00:14 dist-test-slave-tzct krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593614, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:14.271347 22275 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 20:00:14.272332 22275 webserver.cc:460] Webserver started at http://127.8.168.66:45035/ using document root <none> and password file <none>
I20240501 20:00:14.273154 22275 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:14.273221 22275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:14.273466 22275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:14.275810 22275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00"
format_stamp: "Formatted at 2024-05-01 20:00:14 on dist-test-slave-tzct"
server_key: "45f1021b17a140f4110a34c9a7b00ee1"
server_key_iv: "5b0b7b3aff8d01ff900a87d453ba3646"
server_key_version: "encryptionkey@0"
I20240501 20:00:14.276346 22275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00"
format_stamp: "Formatted at 2024-05-01 20:00:14 on dist-test-slave-tzct"
server_key: "45f1021b17a140f4110a34c9a7b00ee1"
server_key_iv: "5b0b7b3aff8d01ff900a87d453ba3646"
server_key_version: "encryptionkey@0"
I20240501 20:00:14.279933 22275 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:00:14.282372 22290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:14.283440 22275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:00:14.283545 22275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00"
format_stamp: "Formatted at 2024-05-01 20:00:14 on dist-test-slave-tzct"
server_key: "45f1021b17a140f4110a34c9a7b00ee1"
server_key_iv: "5b0b7b3aff8d01ff900a87d453ba3646"
server_key_version: "encryptionkey@0"
I20240501 20:00:14.283630 22275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:14.304836 22275 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:14.308070 22275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:14.308298 22275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:14.308898 22275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:14.309801 22275 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:14.309852 22275 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:14.309895 22275 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:14.309918 22275 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:14.318713 22275 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:41379
I20240501 20:00:14.318734 22403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:41379 every 8 connection(s)
I20240501 20:00:14.319643 22275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 20:00:14.326043 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 22275
I20240501 20:00:14.326157 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 20:00:14.326421 8865 external_mini_cluster.cc:1428] Setting key 6fdb28313d8b6ade3b201ee38d9a24cb
May 01 20:00:14 dist-test-slave-tzct krb5kdc[22034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593614, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:14.331828 22118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.321305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:57077 (local address 127.8.168.126:41909)
0501 20:00:14.321440 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:14.321443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:14.322079 (+ 636us) server_negotiation.cc:407] Connection header received
0501 20:00:14.323287 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:14.323290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:14.323352 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:14.323425 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:14.324714 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.325244 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.325856 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.326034 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.328429 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:14.328448 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:14.328450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:14.328474 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:14.329864 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:14.330381 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:14.330385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:14.330386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:14.330441 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:14.330714 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:14.330718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:14.330719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:14.330898 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:14.330988 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:14.331508 (+ 520us) server_negotiation.cc:299] Negotiation successful
0501 20:00:14.331638 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 20:00:14.332399 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.321544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:57077)
0501 20:00:14.321937 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:14.321981 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:14.323076 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:14.323603 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:14.323611 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:14.323997 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:14.324569 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:14.324582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.325358 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.325361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:14.325746 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:14.325753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.325975 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.326711 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:14.326738 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:14.328271 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:14.329992 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:14.329998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:14.330011 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:14.330282 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:14.330525 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:14.330527 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:14.330529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:14.330623 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:14.330988 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:14.330993 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:14.331269 (+ 276us) client_negotiation.cc:769] Sending connection context
0501 20:00:14.331443 (+ 174us) client_negotiation.cc:240] Negotiation successful
0501 20:00:14.331701 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":97,"threads_started":1}
I20240501 20:00:14.333459 22404 heartbeater.cc:347] Connected to a master server at 127.8.168.126:41909
I20240501 20:00:14.333674 22404 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:14.334172 22404 heartbeater.cc:511] Master 127.8.168.126:41909 requested a full tablet report, sending...
I20240501 20:00:14.335175 22079 ts_manager.cc:194] Registered new tserver with Master: b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379)
WARNING: no policy specified for kudu/127.8.168.67@KRBTEST.COM; defaulting to no policy
I20240501 20:00:14.335741 22079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:57077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
I20240501 20:00:14.351678 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:41909
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:14.456799 22409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:14.457016 22409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:14.457080 22409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:14.460127 22409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:14.460184 22409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:14.460281 22409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 20:00:14.464350 22409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:14.465162 22409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:14.465929 22409 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:14.473482 22417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:14.474023 22415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:14.474438 22414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:14.475082 22409 server_base.cc:1034] running on GCE node
I20240501 20:00:14.475447 22409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:14.475966 22409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:14.477130 22409 hybrid_clock.cc:648] HybridClock initialized: now 1714593614477117 us; error 30 us; skew 500 ppm
May 01 20:00:14 dist-test-slave-tzct krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593614, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:14.479790 22409 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 20:00:14.480713 22409 webserver.cc:460] Webserver started at http://127.8.168.67:42549/ using document root <none> and password file <none>
I20240501 20:00:14.481310 22409 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:14.481364 22409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:14.481510 22409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:14.483314 22409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "5d1cc43c44274fd18fddf7c91eca2806"
format_stamp: "Formatted at 2024-05-01 20:00:14 on dist-test-slave-tzct"
server_key: "8eb65fcff5dfa74d214964561f5ef21e"
server_key_iv: "b066f2595a33c03834f8cd1f8b4092d9"
server_key_version: "encryptionkey@0"
I20240501 20:00:14.483745 22409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "5d1cc43c44274fd18fddf7c91eca2806"
format_stamp: "Formatted at 2024-05-01 20:00:14 on dist-test-slave-tzct"
server_key: "8eb65fcff5dfa74d214964561f5ef21e"
server_key_iv: "b066f2595a33c03834f8cd1f8b4092d9"
server_key_version: "encryptionkey@0"
I20240501 20:00:14.487016 22409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 20:00:14.489302 22424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:14.490432 22409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:14.490545 22409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "5d1cc43c44274fd18fddf7c91eca2806"
format_stamp: "Formatted at 2024-05-01 20:00:14 on dist-test-slave-tzct"
server_key: "8eb65fcff5dfa74d214964561f5ef21e"
server_key_iv: "b066f2595a33c03834f8cd1f8b4092d9"
server_key_version: "encryptionkey@0"
I20240501 20:00:14.490643 22409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:14.508347 22409 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:14.511009 22409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:14.511163 22409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:14.511673 22409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:14.512436 22409 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:14.512482 22409 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:14.512524 22409 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:14.512553 22409 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:14.520633 22409 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:41511
I20240501 20:00:14.520679 22537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:41511 every 8 connection(s)
I20240501 20:00:14.521541 22409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 20:00:14.526957 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 22409
I20240501 20:00:14.527081 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 20:00:14.527307 8865 external_mini_cluster.cc:1428] Setting key a49c75e5dff58d670b634e7c3574d834
May 01 20:00:14 dist-test-slave-tzct krb5kdc[22034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593614, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:14.533376 22118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.523209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:55821 (local address 127.8.168.126:41909)
0501 20:00:14.523366 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:14.523370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:14.523981 (+ 611us) server_negotiation.cc:407] Connection header received
0501 20:00:14.525272 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:14.525276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:14.525337 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:14.525410 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:14.526653 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.527356 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.528141 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.528319 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.530481 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:14.530498 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:14.530500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:14.530525 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:14.531716 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:14.532159 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:14.532162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:14.532163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:14.532202 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:14.532412 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:14.532415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:14.532416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:14.532561 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:14.532635 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:14.533151 (+ 516us) server_negotiation.cc:299] Negotiation successful
0501 20:00:14.533253 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20240501 20:00:14.533998 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.523444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:55821)
0501 20:00:14.523849 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:14.523884 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:14.525052 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:14.525575 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:14.525582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:14.525952 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:14.526507 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:14.526520 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.527504 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.527507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:14.528008 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:14.528018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.528205 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.528921 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:14.528945 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:14.530339 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:14.531846 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:14.531850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:14.531860 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:14.532080 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:14.532268 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:14.532270 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:14.532272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:14.532344 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:14.532634 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:14.532638 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:14.532918 (+ 280us) client_negotiation.cc:769] Sending connection context
0501 20:00:14.533083 (+ 165us) client_negotiation.cc:240] Negotiation successful
0501 20:00:14.533299 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
I20240501 20:00:14.535017 22538 heartbeater.cc:347] Connected to a master server at 127.8.168.126:41909
I20240501 20:00:14.535203 22538 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:14.535692 22538 heartbeater.cc:511] Master 127.8.168.126:41909 requested a full tablet report, sending...
I20240501 20:00:14.536659 22079 ts_manager.cc:194] Registered new tserver with Master: 5d1cc43c44274fd18fddf7c91eca2806 (127.8.168.67:41511)
I20240501 20:00:14.537267 22079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:55821
I20240501 20:00:14.541182 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 20:00:14.548951 22118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.543031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.8.168.126:41909)
0501 20:00:14.543167 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:14.543170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:14.543351 (+ 181us) server_negotiation.cc:407] Connection header received
0501 20:00:14.543444 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:14.543446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:14.543486 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:14.543557 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:14.544369 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.544901 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.545507 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.545631 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.546579 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:14.546593 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:14.546595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:14.546612 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:14.547750 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:14.548095 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:14.548098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:14.548099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:14.548143 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:14.548331 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:14.548333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:14.548334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:14.548485 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:14.548582 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:14.548739 (+ 157us) server_negotiation.cc:299] Negotiation successful
0501 20:00:14.548836 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20240501 20:00:14.551491 22079 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:00:14.554051 22079 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:00:14.566011 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.561801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:44835 (local address 127.0.0.1:45950)
0501 20:00:14.562352 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:14.562384 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:14.562752 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:14.563367 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:14.563372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:14.563394 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:14.563681 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:14.563690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.564760 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.564764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:14.565532 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:14.565541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.565638 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.565695 (+ 57us) client_negotiation.cc:769] Sending connection context
0501 20:00:14.565801 (+ 106us) client_negotiation.cc:240] Negotiation successful
0501 20:00:14.565894 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":77,"threads_started":1}
I20240501 20:00:14.566730 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.562463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:41379 (local address 127.0.0.1:43128)
0501 20:00:14.563081 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:14.563093 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:14.563223 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:14.563801 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:14.563804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:14.563822 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:14.564047 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:14.564054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.565378 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.565381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:14.566339 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:14.566350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.566460 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.566476 (+ 16us) client_negotiation.cc:769] Sending connection context
0501 20:00:14.566524 (+ 48us) client_negotiation.cc:240] Negotiation successful
0501 20:00:14.566579 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":88,"threads_started":1}
I20240501 20:00:14.566730 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.561872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:41511 (local address 127.0.0.1:38710)
0501 20:00:14.562442 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:14.562454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:14.562571 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:14.563673 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:14.563677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:14.563694 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:14.563894 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:14.563901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.565377 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.565381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:14.566351 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:14.566358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.566459 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.566476 (+ 17us) client_negotiation.cc:769] Sending connection context
0501 20:00:14.566524 (+ 48us) client_negotiation.cc:240] Negotiation successful
0501 20:00:14.566579 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":91,"threads_started":1}
I20240501 20:00:14.567481 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.561922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.8.168.65:44835)
0501 20:00:14.562230 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:14.562237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:14.562672 (+ 435us) server_negotiation.cc:407] Connection header received
0501 20:00:14.562949 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:14.562955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:14.563143 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:14.563307 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:14.563811 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.564652 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.566317 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.567030 (+ 713us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.567173 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:14.567245 (+ 72us) server_negotiation.cc:299] Negotiation successful
0501 20:00:14.567335 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":108,"threads_started":1}
I20240501 20:00:14.567478 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.562623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.8.168.66:41379)
0501 20:00:14.562944 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:14.562950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:14.563161 (+ 211us) server_negotiation.cc:407] Connection header received
0501 20:00:14.563374 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:14.563379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:14.563534 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:14.563643 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:14.564160 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.565265 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.566484 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.567123 (+ 639us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.567201 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:14.567253 (+ 52us) server_negotiation.cc:299] Negotiation successful
0501 20:00:14.567335 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":99,"threads_started":1}
I20240501 20:00:14.567736 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.562950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38710 (local address 127.8.168.67:41511)
0501 20:00:14.563297 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:14.563303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:14.563323 (+ 20us) server_negotiation.cc:407] Connection header received
0501 20:00:14.563403 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:14.563407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:14.563545 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:14.563673 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:14.564001 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.565248 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.567039 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.567389 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.567469 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:14.567524 (+ 55us) server_negotiation.cc:299] Negotiation successful
0501 20:00:14.567601 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20240501 20:00:14.569837 22204 tablet_service.cc:1467] Processing CreateTablet for tablet ace7ddb86c374794aef468a771e36132 (DEFAULT_TABLE table=test-table [id=d872bf33074b4c82a5b2bb522642d1e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:14.569866 22338 tablet_service.cc:1467] Processing CreateTablet for tablet ace7ddb86c374794aef468a771e36132 (DEFAULT_TABLE table=test-table [id=d872bf33074b4c82a5b2bb522642d1e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:14.569867 22472 tablet_service.cc:1467] Processing CreateTablet for tablet ace7ddb86c374794aef468a771e36132 (DEFAULT_TABLE table=test-table [id=d872bf33074b4c82a5b2bb522642d1e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:14.570680 22204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace7ddb86c374794aef468a771e36132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:14.570701 22472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace7ddb86c374794aef468a771e36132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:14.570703 22338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace7ddb86c374794aef468a771e36132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:14.575917 22555 tablet_bootstrap.cc:492] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: Bootstrap starting.
I20240501 20:00:14.577416 22556 tablet_bootstrap.cc:492] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806: Bootstrap starting.
I20240501 20:00:14.577389 22557 tablet_bootstrap.cc:492] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00: Bootstrap starting.
I20240501 20:00:14.578933 22555 tablet_bootstrap.cc:654] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:14.578944 22556 tablet_bootstrap.cc:654] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:14.579211 22557 tablet_bootstrap.cc:654] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:14.579679 22556 log.cc:824] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:14.579779 22555 log.cc:824] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:14.580001 22557 log.cc:824] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:14.581279 22556 tablet_bootstrap.cc:492] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806: No bootstrap required, opened a new log
I20240501 20:00:14.581437 22556 ts_tablet_manager.cc:1367] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:00:14.581527 22557 tablet_bootstrap.cc:492] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00: No bootstrap required, opened a new log
I20240501 20:00:14.581701 22557 ts_tablet_manager.cc:1367] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:00:14.582669 22555 tablet_bootstrap.cc:492] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: No bootstrap required, opened a new log
I20240501 20:00:14.582855 22555 ts_tablet_manager.cc:1367] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:00:14.584502 22557 raft_consensus.cc:348] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } }
I20240501 20:00:14.584671 22557 raft_consensus.cc:374] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:14.584748 22557 raft_consensus.cc:729] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6f662d52dbc4fd8bf0fdcc9aaa44c00, State: Initialized, Role: FOLLOWER
I20240501 20:00:14.584869 22556 raft_consensus.cc:348] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } }
I20240501 20:00:14.585068 22556 raft_consensus.cc:374] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:14.585119 22556 raft_consensus.cc:729] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1cc43c44274fd18fddf7c91eca2806, State: Initialized, Role: FOLLOWER
I20240501 20:00:14.585160 22557 consensus_queue.cc:260] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } }
I20240501 20:00:14.585556 22556 consensus_queue.cc:260] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } }
I20240501 20:00:14.585773 22404 heartbeater.cc:503] Master 127.8.168.126:41909 was elected leader, sending a full tablet report...
I20240501 20:00:14.585992 22557 ts_tablet_manager.cc:1398] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 20:00:14.586050 22555 raft_consensus.cc:348] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } }
I20240501 20:00:14.586215 22555 raft_consensus.cc:374] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:14.586258 22555 raft_consensus.cc:729] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee8e00be6684bce8007800fcb7e9df5, State: Initialized, Role: FOLLOWER
I20240501 20:00:14.586325 22556 ts_tablet_manager.cc:1398] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:00:14.586480 22538 heartbeater.cc:503] Master 127.8.168.126:41909 was elected leader, sending a full tablet report...
I20240501 20:00:14.586638 22555 consensus_queue.cc:260] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } }
I20240501 20:00:14.587337 22555 ts_tablet_manager.cc:1398] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 20:00:14.587641 22270 heartbeater.cc:503] Master 127.8.168.126:41909 was elected leader, sending a full tablet report...
W20240501 20:00:14.606878 22271 tablet.cc:2369] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:00:14.684980 22562 raft_consensus.cc:482] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:14.685169 22562 raft_consensus.cc:504] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } }
I20240501 20:00:14.686266 22562 leader_election.cc:288] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ee8e00be6684bce8007800fcb7e9df5 (127.8.168.65:44835), b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379)
I20240501 20:00:14.689222 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.686478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:44835 (local address 127.8.168.67:58309)
0501 20:00:14.686658 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:14.686672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:14.686770 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:14.687068 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:14.687071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:14.687092 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:14.687350 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:14.687358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.688145 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.688148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:14.688837 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:14.688846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.688935 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.688956 (+ 21us) client_negotiation.cc:769] Sending connection context
0501 20:00:14.689004 (+ 48us) client_negotiation.cc:240] Negotiation successful
0501 20:00:14.689058 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20240501 20:00:14.689257 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.686510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:41379 (local address 127.8.168.67:53911)
0501 20:00:14.686841 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:14.686854 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:14.686949 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:14.687227 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:14.687229 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:14.687244 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:14.687527 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:14.687536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.688234 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.688237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:14.688872 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:14.688880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.688988 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.689006 (+ 18us) client_negotiation.cc:769] Sending connection context
0501 20:00:14.689052 (+ 46us) client_negotiation.cc:240] Negotiation successful
0501 20:00:14.689104 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":114,"threads_started":1}
I20240501 20:00:14.689657 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.686568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:58309 (local address 127.8.168.65:44835)
0501 20:00:14.686678 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:14.686681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:14.686765 (+ 84us) server_negotiation.cc:407] Connection header received
0501 20:00:14.686899 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:14.686903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:14.686958 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:14.687030 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:14.687456 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.688040 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.688942 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.689399 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.689439 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:14.689489 (+ 50us) server_negotiation.cc:299] Negotiation successful
0501 20:00:14.689549 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240501 20:00:14.689690 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.686610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:53911 (local address 127.8.168.66:41379)
0501 20:00:14.686794 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:14.686798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:14.686943 (+ 145us) server_negotiation.cc:407] Connection header received
0501 20:00:14.687059 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:14.687062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:14.687134 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:14.687219 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:14.687646 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.688117 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.688987 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.689460 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.689494 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:14.689540 (+ 46us) server_negotiation.cc:299] Negotiation successful
0501 20:00:14.689596 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20240501 20:00:14.690161 22224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace7ddb86c374794aef468a771e36132" candidate_uuid: "5d1cc43c44274fd18fddf7c91eca2806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee8e00be6684bce8007800fcb7e9df5" is_pre_election: true
I20240501 20:00:14.690191 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace7ddb86c374794aef468a771e36132" candidate_uuid: "5d1cc43c44274fd18fddf7c91eca2806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" is_pre_election: true
I20240501 20:00:14.690419 22358 raft_consensus.cc:2454] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d1cc43c44274fd18fddf7c91eca2806 in term 0.
I20240501 20:00:14.690419 22224 raft_consensus.cc:2454] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d1cc43c44274fd18fddf7c91eca2806 in term 0.
I20240501 20:00:14.690850 22428 leader_election.cc:302] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d1cc43c44274fd18fddf7c91eca2806, b6f662d52dbc4fd8bf0fdcc9aaa44c00; no voters:
I20240501 20:00:14.691087 22562 raft_consensus.cc:2781] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:14.691155 22562 raft_consensus.cc:482] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:14.691185 22562 raft_consensus.cc:3037] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:14.692093 22562 raft_consensus.cc:504] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } }
I20240501 20:00:14.692426 22562 leader_election.cc:288] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [CANDIDATE]: Term 1 election: Requested vote from peers 4ee8e00be6684bce8007800fcb7e9df5 (127.8.168.65:44835), b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379)
I20240501 20:00:14.692770 22224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace7ddb86c374794aef468a771e36132" candidate_uuid: "5d1cc43c44274fd18fddf7c91eca2806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee8e00be6684bce8007800fcb7e9df5"
I20240501 20:00:14.692801 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace7ddb86c374794aef468a771e36132" candidate_uuid: "5d1cc43c44274fd18fddf7c91eca2806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00"
I20240501 20:00:14.692905 22358 raft_consensus.cc:3037] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:14.692905 22224 raft_consensus.cc:3037] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:14.694093 22358 raft_consensus.cc:2454] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d1cc43c44274fd18fddf7c91eca2806 in term 1.
I20240501 20:00:14.694088 22224 raft_consensus.cc:2454] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d1cc43c44274fd18fddf7c91eca2806 in term 1.
I20240501 20:00:14.694376 22428 leader_election.cc:302] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d1cc43c44274fd18fddf7c91eca2806, b6f662d52dbc4fd8bf0fdcc9aaa44c00; no voters:
I20240501 20:00:14.694535 22562 raft_consensus.cc:2781] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:14.694733 22562 raft_consensus.cc:686] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [term 1 LEADER]: Becoming Leader. State: Replica: 5d1cc43c44274fd18fddf7c91eca2806, State: Running, Role: LEADER
I20240501 20:00:14.695035 22562 consensus_queue.cc:237] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } }
I20240501 20:00:14.698030 22076 catalog_manager.cc:5484] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d1cc43c44274fd18fddf7c91eca2806 (127.8.168.67). New cstate: current_term: 1 leader_uuid: "5d1cc43c44274fd18fddf7c91eca2806" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:14.718369 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.715559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.8.168.67:41511)
0501 20:00:14.715672 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:14.715675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:14.715688 (+ 13us) server_negotiation.cc:407] Connection header received
0501 20:00:14.715778 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:14.715780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:14.715831 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:14.715926 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:00:14.716205 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.716749 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:14.717400 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:14.717524 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:14.717608 (+ 84us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:00:14.718018 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:00:14.718104 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:14.718200 (+ 96us) server_negotiation.cc:299] Negotiation successful
0501 20:00:14.718242 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240501 20:00:14.723577 22224 raft_consensus.cc:1261] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Refusing update from remote peer 5d1cc43c44274fd18fddf7c91eca2806: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:00:14.723578 22358 raft_consensus.cc:1261] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 1 FOLLOWER]: Refusing update from remote peer 5d1cc43c44274fd18fddf7c91eca2806: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:00:14.724273 22565 consensus_queue.cc:1035] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:14.724459 22562 consensus_queue.cc:1035] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:14.731741 22569 mvcc.cc:205] Tried to move back new op lower bound from 7022975445901172736 to 7022975445793386496. Current Snapshot: MvccSnapshot[applied={T|T < 7022975445901172736}]
I20240501 20:00:14.732501 22570 mvcc.cc:205] Tried to move back new op lower bound from 7022975445901172736 to 7022975445793386496. Current Snapshot: MvccSnapshot[applied={T|T < 7022975445901172736}]
I20240501 20:00:14.739449 22079 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42326:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:00:14.739609 22079 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42326:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:00:14.741923 22079 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12: Sending DeleteTablet for 3 replicas of tablet ace7ddb86c374794aef468a771e36132
I20240501 20:00:14.742743 22472 tablet_service.cc:1514] Processing DeleteTablet for tablet ace7ddb86c374794aef468a771e36132 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:00:14 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:38710
I20240501 20:00:14.742733 22204 tablet_service.cc:1514] Processing DeleteTablet for tablet ace7ddb86c374794aef468a771e36132 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:00:14 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:45950
I20240501 20:00:14.742892 22338 tablet_service.cc:1514] Processing DeleteTablet for tablet ace7ddb86c374794aef468a771e36132 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:00:14 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:43128
I20240501 20:00:14.743394 22578 tablet_replica.cc:330] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806: stopping tablet replica
I20240501 20:00:14.743430 22579 tablet_replica.cc:330] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: stopping tablet replica
I20240501 20:00:14.743453 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 22141
I20240501 20:00:14.743680 22578 raft_consensus.cc:2229] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:00:14.743680 22579 raft_consensus.cc:2229] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:14.744014 22578 raft_consensus.cc:2258] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:14.744094 22580 tablet_replica.cc:330] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00: stopping tablet replica
I20240501 20:00:14.744318 22580 raft_consensus.cc:2229] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:14.744611 22580 raft_consensus.cc:2258] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:14.745807 22580 ts_tablet_manager.cc:1875] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:00:14.745807 22578 ts_tablet_manager.cc:1875] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:00:14.748219 22580 ts_tablet_manager.cc:1888] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:00:14.748219 22578 ts_tablet_manager.cc:1888] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:00:14.748308 22578 log.cc:1196] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/ace7ddb86c374794aef468a771e36132
I20240501 20:00:14.748356 22580 log.cc:1196] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/ace7ddb86c374794aef468a771e36132
I20240501 20:00:14.748656 22578 ts_tablet_manager.cc:1909] T ace7ddb86c374794aef468a771e36132 P 5d1cc43c44274fd18fddf7c91eca2806: Deleting consensus metadata
I20240501 20:00:14.748677 22580 ts_tablet_manager.cc:1909] T ace7ddb86c374794aef468a771e36132 P b6f662d52dbc4fd8bf0fdcc9aaa44c00: Deleting consensus metadata
I20240501 20:00:14.749830 22066 catalog_manager.cc:4830] TS 5d1cc43c44274fd18fddf7c91eca2806 (127.8.168.67:41511): tablet ace7ddb86c374794aef468a771e36132 (table test-table [id=d872bf33074b4c82a5b2bb522642d1e6]) successfully deleted
I20240501 20:00:14.750025 22066 catalog_manager.cc:4830] TS b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379): tablet ace7ddb86c374794aef468a771e36132 (table test-table [id=d872bf33074b4c82a5b2bb522642d1e6]) successfully deleted
W20240501 20:00:14.752111 22065 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.168.65:44835 (error 108)
I20240501 20:00:14.753086 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:14.752825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:44835 (local address 127.0.0.1:45960)
0501 20:00:14.752957 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:14.753010 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20240501 20:00:14.753212 22065 catalog_manager.cc:4573] TS 4ee8e00be6684bce8007800fcb7e9df5 (127.8.168.65:44835): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ace7ddb86c374794aef468a771e36132: Network error: Client connection negotiation failed: client connection to 127.8.168.65:44835: connect: Connection refused (error 111)
I20240501 20:00:14.753304 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 22275
I20240501 20:00:14.758472 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 22409
I20240501 20:00:14.763774 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 22047
I20240501 20:00:14.768929 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:41909
--webserver_interface=127.8.168.126
--webserver_port=39257
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:41909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:14.873095 22582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:14.873339 22582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:14.873400 22582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:14.876438 22582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:00:14.876506 22582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:14.876528 22582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:14.876543 22582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:00:14.876559 22582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:00:14.880673 22582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:41909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:41909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=39257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:14.881575 22582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:14.882395 22582 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:14.894119 22588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:14.894336 22591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:14.894572 22589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:14.895222 22582 server_base.cc:1034] running on GCE node
I20240501 20:00:14.895817 22582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:14.896839 22582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:14.898020 22582 hybrid_clock.cc:648] HybridClock initialized: now 1714593614897987 us; error 47 us; skew 500 ppm
May 01 20:00:14 dist-test-slave-tzct krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593614, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:14.900945 22582 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:00:14.901970 22582 webserver.cc:460] Webserver started at http://127.8.168.126:39257/ using document root <none> and password file <none>
I20240501 20:00:14.902531 22582 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:14.902609 22582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:14.906334 22582 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:14.908317 22598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:14.909397 22582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:14.909536 22582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "105abcddc47f43b3a2dcc204988e5c12"
format_stamp: "Formatted at 2024-05-01 20:00:13 on dist-test-slave-tzct"
server_key: "a5ed3b11e9c81c1afb542785a1e3c684"
server_key_iv: "67ef9404aad49452738f6c629dd3a27e"
server_key_version: "encryptionkey@0"
I20240501 20:00:14.910061 22582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:14.925761 22582 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:14.928547 22582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:14.928750 22582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:14.935554 22582 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:41909
I20240501 20:00:14.935577 22650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:41909 every 8 connection(s)
I20240501 20:00:14.936965 22582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:00:14.941017 22651 sys_catalog.cc:263] Verifying existing consensus state
I20240501 20:00:14.942658 22651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12: Bootstrap starting.
I20240501 20:00:14.944593 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 22582
I20240501 20:00:14.945101 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:44835
--local_ip_for_outbound_sockets=127.8.168.65
--tserver_master_addrs=127.8.168.126:41909
--webserver_port=43869
--webserver_interface=127.8.168.65
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 20:00:14.959456 22651 log.cc:824] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:14.967854 22651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:00:14.968214 22651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12: Bootstrap complete.
I20240501 20:00:14.971570 22651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } }
I20240501 20:00:14.971905 22651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105abcddc47f43b3a2dcc204988e5c12, State: Initialized, Role: FOLLOWER
I20240501 20:00:14.972384 22651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } }
I20240501 20:00:14.972499 22651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:00:14.972558 22651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:00:14.972651 22651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:14.974267 22651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } }
I20240501 20:00:14.974586 22651 leader_election.cc:302] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 105abcddc47f43b3a2dcc204988e5c12; no voters:
I20240501 20:00:14.974859 22651 leader_election.cc:288] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240501 20:00:14.974960 22655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:14.975163 22655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 2 LEADER]: Becoming Leader. State: Replica: 105abcddc47f43b3a2dcc204988e5c12, State: Running, Role: LEADER
I20240501 20:00:14.975504 22655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } }
I20240501 20:00:14.976004 22651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:00:14.976927 22656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "105abcddc47f43b3a2dcc204988e5c12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } } }
I20240501 20:00:14.977046 22656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:14.977286 22657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 105abcddc47f43b3a2dcc204988e5c12. Latest consensus state: current_term: 2 leader_uuid: "105abcddc47f43b3a2dcc204988e5c12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } } }
I20240501 20:00:14.977352 22657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:14.977389 22664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:00:14.982409 22664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:00:14.983309 22664 catalog_manager.cc:1259] Loaded cluster ID: 383f7000892044a9ae7c31e097a2321d
I20240501 20:00:14.983387 22664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:00:14.984619 22664 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20240501 20:00:14.986282 22664 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20240501 20:00:14.990779 22664 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:00:14.991605 22664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12: Loaded TSK: 0
I20240501 20:00:14.992642 22664 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240501 20:00:15.068023 22653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:15.068295 22653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:15.068374 22653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:15.071617 22653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:15.071691 22653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:15.071799 22653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 20:00:15.076035 22653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:44835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=43869
--tserver_master_addrs=127.8.168.126:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:15.077077 22653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:15.078020 22653 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:15.086195 22678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:15.086678 22679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:15.087056 22681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:15.087718 22653 server_base.cc:1034] running on GCE node
I20240501 20:00:15.088155 22653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:15.088801 22653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:15.089984 22653 hybrid_clock.cc:648] HybridClock initialized: now 1714593615089961 us; error 42 us; skew 500 ppm
May 01 20:00:15 dist-test-slave-tzct krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:15.092875 22653 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 20:00:15.093833 22653 webserver.cc:460] Webserver started at http://127.8.168.65:43869/ using document root <none> and password file <none>
I20240501 20:00:15.094368 22653 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:15.094442 22653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:15.097955 22653 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:15.100135 22688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:15.101251 22653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:15.101401 22653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "4ee8e00be6684bce8007800fcb7e9df5"
format_stamp: "Formatted at 2024-05-01 20:00:14 on dist-test-slave-tzct"
server_key: "584ed12a423040d313e424f0aac5be66"
server_key_iv: "142efdf649458d55ac5e978078b3f91d"
server_key_version: "encryptionkey@0"
I20240501 20:00:15.101828 22653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:15.121582 22653 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:15.124812 22653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:15.124981 22653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:15.125559 22653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:15.126605 22695 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 20:00:15.130789 22653 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 20:00:15.130865 22653 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20240501 20:00:15.130936 22653 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 20:00:15.132953 22653 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 20:00:15.133005 22653 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:15.133284 22695 tablet_bootstrap.cc:492] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: Bootstrap starting.
I20240501 20:00:15.142938 22653 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:44835
I20240501 20:00:15.143180 22802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:44835 every 8 connection(s)
I20240501 20:00:15.144026 22653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 20:00:15.151537 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 22653
I20240501 20:00:15.152032 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:41379
--local_ip_for_outbound_sockets=127.8.168.66
--tserver_master_addrs=127.8.168.126:41909
--webserver_port=45035
--webserver_interface=127.8.168.66
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 20:00:15 dist-test-slave-tzct krb5kdc[22034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:15.153824 22695 log.cc:824] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:15.159310 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.145893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:47893)
0501 20:00:15.146569 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:15.146612 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:15.147405 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:15.148537 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:15.148548 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:15.149026 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:15.149598 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:15.149613 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.150710 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.150713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:15.151096 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:15.151104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.151277 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.152414 (+ 1137us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:15.152454 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:15.154208 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:15.156659 (+ 2451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:15.156665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:15.156675 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:15.156970 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:15.157262 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:15.157264 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:15.157266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:15.157347 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:15.157861 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:15.157867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:15.158096 (+ 229us) client_negotiation.cc:769] Sending connection context
0501 20:00:15.158262 (+ 166us) client_negotiation.cc:240] Negotiation successful
0501 20:00:15.158558 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":114,"threads_started":1}
I20240501 20:00:15.159296 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.145886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:47893 (local address 127.8.168.126:41909)
0501 20:00:15.146301 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:15.146312 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:15.147671 (+ 1359us) server_negotiation.cc:407] Connection header received
0501 20:00:15.147892 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:15.147902 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:15.148162 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:15.148459 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:15.149767 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.150575 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.151205 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.152056 (+ 851us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.154437 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:15.154463 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:15.154478 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:15.154525 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:15.156534 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:15.157081 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:15.157086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:15.157089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:15.157170 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:15.157446 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:15.157448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:15.157449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:15.157747 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:15.157877 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:15.158422 (+ 545us) server_negotiation.cc:299] Negotiation successful
0501 20:00:15.158669 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":99,"threads_started":1}
I20240501 20:00:15.161373 22803 heartbeater.cc:347] Connected to a master server at 127.8.168.126:41909
I20240501 20:00:15.161610 22803 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:15.162120 22803 heartbeater.cc:511] Master 127.8.168.126:41909 requested a full tablet report, sending...
I20240501 20:00:15.162891 22695 tablet_bootstrap.cc:492] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:00:15.163291 22695 tablet_bootstrap.cc:492] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: Bootstrap complete.
I20240501 20:00:15.164428 22614 ts_manager.cc:194] Registered new tserver with Master: 4ee8e00be6684bce8007800fcb7e9df5 (127.8.168.65:44835)
I20240501 20:00:15.164573 22695 ts_tablet_manager.cc:1367] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.004s
I20240501 20:00:15.167290 22695 raft_consensus.cc:348] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } }
I20240501 20:00:15.167735 22695 raft_consensus.cc:729] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee8e00be6684bce8007800fcb7e9df5, State: Initialized, Role: FOLLOWER
I20240501 20:00:15.167809 22614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:47893
I20240501 20:00:15.168233 22695 consensus_queue.cc:260] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } }
I20240501 20:00:15.169155 22695 ts_tablet_manager.cc:1398] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20240501 20:00:15.170341 22803 heartbeater.cc:503] Master 127.8.168.126:41909 was elected leader, sending a full tablet report...
I20240501 20:00:15.172873 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.167236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:44835 (local address 127.0.0.1:45974)
0501 20:00:15.168200 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:15.168236 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:15.168757 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:15.169983 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:15.169988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:15.170014 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:15.170289 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:15.170298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.171541 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.171545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:15.172431 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:15.172442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.172540 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.172578 (+ 38us) client_negotiation.cc:769] Sending connection context
0501 20:00:15.172639 (+ 61us) client_negotiation.cc:240] Negotiation successful
0501 20:00:15.172712 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":133,"threads_started":1}
I20240501 20:00:15.173422 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.167326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.8.168.65:44835)
0501 20:00:15.168092 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:15.168099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:15.168682 (+ 583us) server_negotiation.cc:407] Connection header received
0501 20:00:15.169600 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:15.169606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:15.169783 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:15.169947 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:15.170427 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.171420 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.172557 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.173048 (+ 491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.173157 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:15.173224 (+ 67us) server_negotiation.cc:299] Negotiation successful
0501 20:00:15.173294 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":104,"threads_started":1}
I20240501 20:00:15.174270 22737 tablet_service.cc:1514] Processing DeleteTablet for tablet ace7ddb86c374794aef468a771e36132 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:00:14 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:45974
I20240501 20:00:15.175120 22812 tablet_replica.cc:330] stopping tablet replica
I20240501 20:00:15.175354 22812 raft_consensus.cc:2229] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:15.175580 22812 raft_consensus.cc:2258] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:15.176537 22812 ts_tablet_manager.cc:1875] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:00:15.179749 22812 ts_tablet_manager.cc:1888] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:00:15.179837 22812 log.cc:1196] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal/wals/ace7ddb86c374794aef468a771e36132
I20240501 20:00:15.180090 22812 ts_tablet_manager.cc:1909] T ace7ddb86c374794aef468a771e36132 P 4ee8e00be6684bce8007800fcb7e9df5: Deleting consensus metadata
I20240501 20:00:15.181818 22601 catalog_manager.cc:4830] TS 4ee8e00be6684bce8007800fcb7e9df5 (127.8.168.65:44835): tablet ace7ddb86c374794aef468a771e36132 (table test-table [id=d872bf33074b4c82a5b2bb522642d1e6]) successfully deleted
W20240501 20:00:15.265482 22807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:15.265736 22807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:15.265810 22807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:15.268914 22807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:15.268980 22807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:15.269078 22807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 20:00:15.273157 22807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:41379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=45035
--tserver_master_addrs=127.8.168.126:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:15.274125 22807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:15.274981 22807 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:15.284371 22817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:15.284564 22820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:15.285094 22818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:15.285617 22807 server_base.cc:1034] running on GCE node
I20240501 20:00:15.286031 22807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:15.286638 22807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:15.287777 22807 hybrid_clock.cc:648] HybridClock initialized: now 1714593615287757 us; error 35 us; skew 500 ppm
May 01 20:00:15 dist-test-slave-tzct krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:15.290452 22807 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 20:00:15.291378 22807 webserver.cc:460] Webserver started at http://127.8.168.66:45035/ using document root <none> and password file <none>
I20240501 20:00:15.291887 22807 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:15.291952 22807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:15.295480 22807 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:15.297463 22827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:15.298523 22807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:15.298628 22807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00"
format_stamp: "Formatted at 2024-05-01 20:00:14 on dist-test-slave-tzct"
server_key: "45f1021b17a140f4110a34c9a7b00ee1"
server_key_iv: "5b0b7b3aff8d01ff900a87d453ba3646"
server_key_version: "encryptionkey@0"
I20240501 20:00:15.299049 22807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:15.313946 22807 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:15.316958 22807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:15.317131 22807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:15.317705 22807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:15.318514 22807 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:15.318572 22807 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:15.318629 22807 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:15.318655 22807 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:15.327168 22807 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:41379
I20240501 20:00:15.327204 22940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:41379 every 8 connection(s)
I20240501 20:00:15.328236 22807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 20:00:15.329311 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 22807
I20240501 20:00:15.329733 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:41511
--local_ip_for_outbound_sockets=127.8.168.67
--tserver_master_addrs=127.8.168.126:41909
--webserver_port=42549
--webserver_interface=127.8.168.67
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 20:00:15 dist-test-slave-tzct krb5kdc[22034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:15.341998 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.331321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:34885 (local address 127.8.168.126:41909)
0501 20:00:15.331490 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:15.331495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:15.331515 (+ 20us) server_negotiation.cc:407] Connection header received
0501 20:00:15.332507 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:15.332510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:15.332556 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:15.332634 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:15.334442 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.334956 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.335804 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.335952 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.338495 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:15.338522 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:15.338525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:15.338559 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:15.340062 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:15.340535 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:15.340539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:15.340541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:15.340597 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:15.340892 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:15.340896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:15.340898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:15.341095 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:15.341197 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:15.341681 (+ 484us) server_negotiation.cc:299] Negotiation successful
0501 20:00:15.341819 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20240501 20:00:15.342588 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.330419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:34885)
0501 20:00:15.330866 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:15.330912 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:15.332263 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:15.332876 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:15.332888 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:15.333436 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:15.334267 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:15.334286 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.335081 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.335085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:15.335697 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:15.335708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.335958 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.336617 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:15.336653 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:15.338298 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:15.340179 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:15.340184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:15.340193 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:15.340440 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:15.340685 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:15.340688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:15.340689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:15.340808 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:15.341184 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:15.341188 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:15.341432 (+ 244us) client_negotiation.cc:769] Sending connection context
0501 20:00:15.341607 (+ 175us) client_negotiation.cc:240] Negotiation successful
0501 20:00:15.341868 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":99,"threads_started":1}
I20240501 20:00:15.343715 22941 heartbeater.cc:347] Connected to a master server at 127.8.168.126:41909
I20240501 20:00:15.343935 22941 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:15.344456 22941 heartbeater.cc:511] Master 127.8.168.126:41909 requested a full tablet report, sending...
I20240501 20:00:15.345474 22614 ts_manager.cc:194] Registered new tserver with Master: b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379)
I20240501 20:00:15.346037 22614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:34885
W20240501 20:00:15.436762 22944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:15.437014 22944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:15.437083 22944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:15.440188 22944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:15.440259 22944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:15.440352 22944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 20:00:15.444453 22944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:41511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=42549
--tserver_master_addrs=127.8.168.126:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:15.445351 22944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:15.446194 22944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:15.454327 22952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:15.454710 22949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:15.455142 22950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:15.455726 22944 server_base.cc:1034] running on GCE node
I20240501 20:00:15.456138 22944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:15.456701 22944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:15.457893 22944 hybrid_clock.cc:648] HybridClock initialized: now 1714593615457869 us; error 36 us; skew 500 ppm
May 01 20:00:15 dist-test-slave-tzct krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:15.460474 22944 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 20:00:15.461467 22944 webserver.cc:460] Webserver started at http://127.8.168.67:42549/ using document root <none> and password file <none>
I20240501 20:00:15.462000 22944 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:15.462069 22944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:15.465588 22944 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:15.467494 22959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:15.468510 22944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:15.468613 22944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "5d1cc43c44274fd18fddf7c91eca2806"
format_stamp: "Formatted at 2024-05-01 20:00:14 on dist-test-slave-tzct"
server_key: "8eb65fcff5dfa74d214964561f5ef21e"
server_key_iv: "b066f2595a33c03834f8cd1f8b4092d9"
server_key_version: "encryptionkey@0"
I20240501 20:00:15.469069 22944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:15.488651 22944 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:15.491451 22944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:15.491616 22944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:15.492174 22944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:15.492990 22944 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:15.493037 22944 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:15.493075 22944 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:15.493098 22944 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:15.501590 22944 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:41511
I20240501 20:00:15.501636 23072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:41511 every 8 connection(s)
I20240501 20:00:15.502714 22944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 20:00:15.505519 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 22944
May 01 20:00:15 dist-test-slave-tzct krb5kdc[22034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:15.514063 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.506213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.8.168.126:41909)
0501 20:00:15.506488 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:15.506492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:15.506609 (+ 117us) server_negotiation.cc:407] Connection header received
0501 20:00:15.506739 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:15.506742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:15.506786 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:15.506869 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:15.507921 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.508493 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.509203 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.509341 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.510582 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:15.510601 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:15.510604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:15.510627 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:15.512566 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:15.513001 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:15.513004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:15.513005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:15.513050 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:15.513266 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:15.513268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:15.513269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:15.513427 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:15.513635 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:15.513781 (+ 146us) server_negotiation.cc:299] Negotiation successful
0501 20:00:15.513898 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"spinlock_wait_cycles":11648,"thread_start_us":64,"threads_started":1}
I20240501 20:00:15.515786 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.504481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:45479 (local address 127.8.168.126:41909)
0501 20:00:15.504629 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:15.504633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:15.505432 (+ 799us) server_negotiation.cc:407] Connection header received
0501 20:00:15.506924 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:15.506927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:15.506992 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:15.507077 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:15.508572 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.509086 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.509815 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.509936 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.512029 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:15.512045 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:15.512048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:15.512071 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:15.513487 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:15.514234 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:15.514238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:15.514240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:15.514289 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:15.514659 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:15.514662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:15.514664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:15.514831 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:15.514931 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:15.515498 (+ 567us) server_negotiation.cc:299] Negotiation successful
0501 20:00:15.515616 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42,"spinlock_wait_cycles":35584}
I20240501 20:00:15.516548 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.504810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:45479)
0501 20:00:15.505252 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:15.505301 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:15.506714 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:15.507282 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:15.507295 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:15.507807 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:15.508438 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:15.508452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.509208 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.509211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:15.509690 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:15.509702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.509959 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.510494 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:15.510534 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:15.511896 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:15.513611 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:15.513617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:15.513628 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:15.514005 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:15.514404 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:15.514407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:15.514408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:15.514507 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:15.514947 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:15.514954 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:15.515251 (+ 297us) client_negotiation.cc:769] Sending connection context
0501 20:00:15.515450 (+ 199us) client_negotiation.cc:240] Negotiation successful
0501 20:00:15.515734 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":98,"threads_started":1}
I20240501 20:00:15.517624 23073 heartbeater.cc:347] Connected to a master server at 127.8.168.126:41909
I20240501 20:00:15.517838 23073 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:15.518332 23073 heartbeater.cc:511] Master 127.8.168.126:41909 requested a full tablet report, sending...
I20240501 20:00:15.519295 22614 ts_manager.cc:194] Registered new tserver with Master: 5d1cc43c44274fd18fddf7c91eca2806 (127.8.168.67:41511)
I20240501 20:00:15.519860 22614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:45479
I20240501 20:00:15.528044 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 20:00:15.531764 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.528658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.8.168.126:41909)
0501 20:00:15.528827 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:15.528830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:15.528981 (+ 151us) server_negotiation.cc:407] Connection header received
0501 20:00:15.529118 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:15.529120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:15.529217 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:15.529318 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:00:15.529632 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.530154 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.530735 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.530889 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.530980 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:00:15.531382 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:00:15.531474 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:15.531597 (+ 123us) server_negotiation.cc:299] Negotiation successful
0501 20:00:15.531653 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240501 20:00:15.532454 22614 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:00:15.535332 22614 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:00:15.543036 22737 tablet_service.cc:1467] Processing CreateTablet for tablet 38a614223ae24bb5ade4c22160fb9d5e (DEFAULT_TABLE table=test-table [id=74546721c60e4601adb4122ffbd9b586]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:15.543615 22737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a614223ae24bb5ade4c22160fb9d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:15.545949 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.542207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:41511 (local address 127.0.0.1:38728)
0501 20:00:15.542447 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:15.542459 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:15.542538 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:15.543296 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:15.543299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:15.543313 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:15.543515 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:15.543521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.544804 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.544808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:15.545561 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:15.545570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.545679 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.545703 (+ 24us) client_negotiation.cc:769] Sending connection context
0501 20:00:15.545752 (+ 49us) client_negotiation.cc:240] Negotiation successful
0501 20:00:15.545811 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":101,"threads_started":1}
I20240501 20:00:15.546224 22695 tablet_bootstrap.cc:492] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5: Bootstrap starting.
I20240501 20:00:15.546084 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.541988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:41379 (local address 127.0.0.1:43130)
0501 20:00:15.542115 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:15.542129 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:15.542222 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:15.543217 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:15.543220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:15.543239 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:15.543490 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:15.543497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.544921 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.544925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:15.545764 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:15.545776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.545859 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.545877 (+ 18us) client_negotiation.cc:769] Sending connection context
0501 20:00:15.545915 (+ 38us) client_negotiation.cc:240] Negotiation successful
0501 20:00:15.545976 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20240501 20:00:15.546669 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.542607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.8.168.67:41511)
0501 20:00:15.542904 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:15.542919 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:15.542942 (+ 23us) server_negotiation.cc:407] Connection header received
0501 20:00:15.543021 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:15.543025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:15.543148 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:15.543647 (+ 499us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:15.543784 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.544485 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.545706 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.546230 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.546361 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:15.546439 (+ 78us) server_negotiation.cc:299] Negotiation successful
0501 20:00:15.546528 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":115,"threads_started":1}
I20240501 20:00:15.547116 22695 tablet_bootstrap.cc:654] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:15.547228 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.542160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.8.168.66:41379)
0501 20:00:15.542749 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:15.542756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:15.542780 (+ 24us) server_negotiation.cc:407] Connection header received
0501 20:00:15.542866 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:15.542872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:15.543060 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:15.543228 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:15.543612 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.544649 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.546345 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.546834 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.546937 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:15.547004 (+ 67us) server_negotiation.cc:299] Negotiation successful
0501 20:00:15.547089 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":203,"threads_started":1}
I20240501 20:00:15.548267 22695 tablet_bootstrap.cc:492] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5: No bootstrap required, opened a new log
I20240501 20:00:15.548367 22695 ts_tablet_manager.cc:1367] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:00:15.548789 22695 raft_consensus.cc:348] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } }
I20240501 20:00:15.548907 22695 raft_consensus.cc:374] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:15.548950 22695 raft_consensus.cc:729] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee8e00be6684bce8007800fcb7e9df5, State: Initialized, Role: FOLLOWER
I20240501 20:00:15.549072 22695 consensus_queue.cc:260] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } }
I20240501 20:00:15.549320 22695 ts_tablet_manager.cc:1398] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 20:00:15.549185 23007 tablet_service.cc:1467] Processing CreateTablet for tablet 38a614223ae24bb5ade4c22160fb9d5e (DEFAULT_TABLE table=test-table [id=74546721c60e4601adb4122ffbd9b586]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:15.549528 22875 tablet_service.cc:1467] Processing CreateTablet for tablet 38a614223ae24bb5ade4c22160fb9d5e (DEFAULT_TABLE table=test-table [id=74546721c60e4601adb4122ffbd9b586]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:15.550372 22875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a614223ae24bb5ade4c22160fb9d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:15.550388 23007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a614223ae24bb5ade4c22160fb9d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:15.555999 23087 tablet_bootstrap.cc:492] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00: Bootstrap starting.
I20240501 20:00:15.556001 23086 tablet_bootstrap.cc:492] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806: Bootstrap starting.
I20240501 20:00:15.558342 23087 tablet_bootstrap.cc:654] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:15.558351 23086 tablet_bootstrap.cc:654] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:15.559042 23087 log.cc:824] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:15.559041 23086 log.cc:824] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:15.560871 23086 tablet_bootstrap.cc:492] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806: No bootstrap required, opened a new log
I20240501 20:00:15.561021 23087 tablet_bootstrap.cc:492] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00: No bootstrap required, opened a new log
I20240501 20:00:15.561067 23086 ts_tablet_manager.cc:1367] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:00:15.561201 23087 ts_tablet_manager.cc:1367] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:00:15.564770 23087 raft_consensus.cc:348] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } }
I20240501 20:00:15.565007 23087 raft_consensus.cc:374] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:15.565058 23087 raft_consensus.cc:729] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6f662d52dbc4fd8bf0fdcc9aaa44c00, State: Initialized, Role: FOLLOWER
I20240501 20:00:15.565534 23087 consensus_queue.cc:260] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } }
I20240501 20:00:15.566274 22941 heartbeater.cc:503] Master 127.8.168.126:41909 was elected leader, sending a full tablet report...
I20240501 20:00:15.566488 23087 ts_tablet_manager.cc:1398] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 20:00:15.566540 23086 raft_consensus.cc:348] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } }
I20240501 20:00:15.566741 23086 raft_consensus.cc:374] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:15.566809 23086 raft_consensus.cc:729] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1cc43c44274fd18fddf7c91eca2806, State: Initialized, Role: FOLLOWER
I20240501 20:00:15.567217 23086 consensus_queue.cc:260] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } }
I20240501 20:00:15.568063 23086 ts_tablet_manager.cc:1398] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:00:15.568420 23073 heartbeater.cc:503] Master 127.8.168.126:41909 was elected leader, sending a full tablet report...
I20240501 20:00:15.575539 23091 raft_consensus.cc:482] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:15.575714 23091 raft_consensus.cc:504] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } }
I20240501 20:00:15.576814 23091 leader_election.cc:288] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379), 4ee8e00be6684bce8007800fcb7e9df5 (127.8.168.65:44835)
W20240501 20:00:15.579128 22942 tablet.cc:2369] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:00:15.579828 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.577008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:44835 (local address 127.8.168.67:56049)
0501 20:00:15.577132 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:15.577161 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:15.577250 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:15.577706 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:15.577710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:15.577730 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:15.577986 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:15.577995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.578813 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.578816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:15.579443 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:15.579459 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.579568 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.579589 (+ 21us) client_negotiation.cc:769] Sending connection context
0501 20:00:15.579638 (+ 49us) client_negotiation.cc:240] Negotiation successful
0501 20:00:15.579695 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20240501 20:00:15.580242 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.577279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:56049 (local address 127.8.168.65:44835)
0501 20:00:15.577474 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:15.577478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:15.577493 (+ 15us) server_negotiation.cc:407] Connection header received
0501 20:00:15.577546 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:15.577548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:15.577600 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:15.577670 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:15.578100 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.578719 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.579565 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.580029 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.580064 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:15.580109 (+ 45us) server_negotiation.cc:299] Negotiation successful
0501 20:00:15.580151 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20240501 20:00:15.580581 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.577008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:41379 (local address 127.8.168.67:51309)
0501 20:00:15.577905 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:15.577918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:15.578001 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:15.578227 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:15.578229 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:15.578241 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:15.578477 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:15.578484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.579346 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.579350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:15.580240 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:15.580250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.580369 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.580386 (+ 17us) client_negotiation.cc:769] Sending connection context
0501 20:00:15.580431 (+ 45us) client_negotiation.cc:240] Negotiation successful
0501 20:00:15.580476 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":299,"thread_start_us":154,"threads_started":1}
I20240501 20:00:15.580906 22757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a614223ae24bb5ade4c22160fb9d5e" candidate_uuid: "5d1cc43c44274fd18fddf7c91eca2806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee8e00be6684bce8007800fcb7e9df5" is_pre_election: true
I20240501 20:00:15.581126 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.577114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:51309 (local address 127.8.168.66:41379)
0501 20:00:15.577284 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:15.577288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:15.577976 (+ 688us) server_negotiation.cc:407] Connection header received
0501 20:00:15.578085 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:15.578088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:15.578138 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:15.578217 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:15.578578 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.579221 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.580373 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.580875 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.580912 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:15.580963 (+ 51us) server_negotiation.cc:299] Negotiation successful
0501 20:00:15.581022 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20240501 20:00:15.581228 22757 raft_consensus.cc:2454] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d1cc43c44274fd18fddf7c91eca2806 in term 0.
I20240501 20:00:15.581684 22962 leader_election.cc:302] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee8e00be6684bce8007800fcb7e9df5, 5d1cc43c44274fd18fddf7c91eca2806; no voters:
I20240501 20:00:15.581693 22895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a614223ae24bb5ade4c22160fb9d5e" candidate_uuid: "5d1cc43c44274fd18fddf7c91eca2806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" is_pre_election: true
I20240501 20:00:15.581943 22895 raft_consensus.cc:2454] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d1cc43c44274fd18fddf7c91eca2806 in term 0.
I20240501 20:00:15.581947 23091 raft_consensus.cc:2781] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:15.582016 23091 raft_consensus.cc:482] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:15.582063 23091 raft_consensus.cc:3037] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:15.583083 23091 raft_consensus.cc:504] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } }
I20240501 20:00:15.583400 23091 leader_election.cc:288] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [CANDIDATE]: Term 1 election: Requested vote from peers b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379), 4ee8e00be6684bce8007800fcb7e9df5 (127.8.168.65:44835)
I20240501 20:00:15.583703 22895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a614223ae24bb5ade4c22160fb9d5e" candidate_uuid: "5d1cc43c44274fd18fddf7c91eca2806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00"
I20240501 20:00:15.583765 22757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a614223ae24bb5ade4c22160fb9d5e" candidate_uuid: "5d1cc43c44274fd18fddf7c91eca2806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee8e00be6684bce8007800fcb7e9df5"
I20240501 20:00:15.583808 22895 raft_consensus.cc:3037] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:15.583864 22757 raft_consensus.cc:3037] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:15.584697 22895 raft_consensus.cc:2454] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d1cc43c44274fd18fddf7c91eca2806 in term 1.
I20240501 20:00:15.585062 22963 leader_election.cc:302] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d1cc43c44274fd18fddf7c91eca2806, b6f662d52dbc4fd8bf0fdcc9aaa44c00; no voters:
I20240501 20:00:15.585139 22757 raft_consensus.cc:2454] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d1cc43c44274fd18fddf7c91eca2806 in term 1.
I20240501 20:00:15.585263 23091 raft_consensus.cc:2781] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:15.585515 23091 raft_consensus.cc:686] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [term 1 LEADER]: Becoming Leader. State: Replica: 5d1cc43c44274fd18fddf7c91eca2806, State: Running, Role: LEADER
I20240501 20:00:15.585999 23091 consensus_queue.cc:237] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } }
I20240501 20:00:15.588929 22613 catalog_manager.cc:5484] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d1cc43c44274fd18fddf7c91eca2806 (127.8.168.67). New cstate: current_term: 1 leader_uuid: "5d1cc43c44274fd18fddf7c91eca2806" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:15.603979 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.601039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.8.168.67:41511)
0501 20:00:15.601150 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:15.601153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:15.601167 (+ 14us) server_negotiation.cc:407] Connection header received
0501 20:00:15.601241 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:15.601243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:15.601303 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:15.601403 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:00:15.601694 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.602210 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:15.602922 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:15.603078 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:15.603164 (+ 86us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:00:15.603643 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:00:15.603735 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:15.603809 (+ 74us) server_negotiation.cc:299] Negotiation successful
0501 20:00:15.603860 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240501 20:00:15.609150 22757 raft_consensus.cc:1261] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Refusing update from remote peer 5d1cc43c44274fd18fddf7c91eca2806: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:00:15.609155 22895 raft_consensus.cc:1261] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 1 FOLLOWER]: Refusing update from remote peer 5d1cc43c44274fd18fddf7c91eca2806: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:00:15.609835 23093 consensus_queue.cc:1035] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:15.609997 23091 consensus_queue.cc:1035] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:15.616585 23096 mvcc.cc:205] Tried to move back new op lower bound from 7022975449528352768 to 7022975449443127296. Current Snapshot: MvccSnapshot[applied={T|T < 7022975449528352768}]
I20240501 20:00:15.619096 23097 mvcc.cc:205] Tried to move back new op lower bound from 7022975449528352768 to 7022975449443127296. Current Snapshot: MvccSnapshot[applied={T|T < 7022975449528352768 or (T in {7022975449528352768})}]
I20240501 20:00:15.623739 22614 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42348:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:00:15.623893 22614 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42348:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:00:15.626147 22614 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12: Sending DeleteTablet for 3 replicas of tablet 38a614223ae24bb5ade4c22160fb9d5e
I20240501 20:00:15.626878 22737 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a614223ae24bb5ade4c22160fb9d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:00:15 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:45974
I20240501 20:00:15.626906 23007 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a614223ae24bb5ade4c22160fb9d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:00:15 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:38728
I20240501 20:00:15.626981 22875 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a614223ae24bb5ade4c22160fb9d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:00:15 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:43130
I20240501 20:00:15.627384 23105 tablet_replica.cc:330] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806: stopping tablet replica
I20240501 20:00:15.627393 22812 tablet_replica.cc:330] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5: stopping tablet replica
I20240501 20:00:15.627524 22812 raft_consensus.cc:2229] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:15.627632 23105 raft_consensus.cc:2229] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:00:15.627662 22812 raft_consensus.cc:2258] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:15.628018 23106 tablet_replica.cc:330] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00: stopping tablet replica
I20240501 20:00:15.628130 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 22653
I20240501 20:00:15.628262 23106 raft_consensus.cc:2229] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:15.628669 23106 raft_consensus.cc:2258] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:15.628870 23105 raft_consensus.cc:2258] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:15.630132 23106 ts_tablet_manager.cc:1875] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:00:15.630137 23105 ts_tablet_manager.cc:1875] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:00:15.632638 23105 ts_tablet_manager.cc:1888] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:00:15.632745 23105 log.cc:1196] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/38a614223ae24bb5ade4c22160fb9d5e
I20240501 20:00:15.632697 23106 ts_tablet_manager.cc:1888] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:00:15.632802 23106 log.cc:1196] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/38a614223ae24bb5ade4c22160fb9d5e
I20240501 20:00:15.633061 23105 ts_tablet_manager.cc:1909] T 38a614223ae24bb5ade4c22160fb9d5e P 5d1cc43c44274fd18fddf7c91eca2806: Deleting consensus metadata
I20240501 20:00:15.633137 23106 ts_tablet_manager.cc:1909] T 38a614223ae24bb5ade4c22160fb9d5e P b6f662d52dbc4fd8bf0fdcc9aaa44c00: Deleting consensus metadata
I20240501 20:00:15.634174 22602 catalog_manager.cc:4830] TS 5d1cc43c44274fd18fddf7c91eca2806 (127.8.168.67:41511): tablet 38a614223ae24bb5ade4c22160fb9d5e (table test-table [id=74546721c60e4601adb4122ffbd9b586]) successfully deleted
I20240501 20:00:15.634387 22602 catalog_manager.cc:4830] TS b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379): tablet 38a614223ae24bb5ade4c22160fb9d5e (table test-table [id=74546721c60e4601adb4122ffbd9b586]) successfully deleted
W20240501 20:00:15.636200 22601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.168.65:44835 (error 108)
I20240501 20:00:15.636704 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 22807
I20240501 20:00:15.638156 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:15.637608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:44835 (local address 127.0.0.1:45982)
0501 20:00:15.638006 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:15.638066 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312}
W20240501 20:00:15.638507 22601 catalog_manager.cc:4573] TS 4ee8e00be6684bce8007800fcb7e9df5 (127.8.168.65:44835): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38a614223ae24bb5ade4c22160fb9d5e: Network error: Client connection negotiation failed: client connection to 127.8.168.65:44835: connect: Connection refused (error 111)
I20240501 20:00:15.642722 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 22944
I20240501 20:00:15.647557 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 22582
I20240501 20:00:15.653080 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:41909
--webserver_interface=127.8.168.126
--webserver_port=39257
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:41909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:15.757980 23108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:15.758219 23108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:15.758276 23108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:15.761332 23108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:00:15.761397 23108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:15.761420 23108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:15.761452 23108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:00:15.761482 23108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:00:15.765635 23108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:41909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:41909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=39257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:15.766541 23108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:15.767366 23108 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:15.774900 23113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:15.774900 23114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:15.775468 23116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:15.776036 23108 server_base.cc:1034] running on GCE node
I20240501 20:00:15.776567 23108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:15.777578 23108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:15.778748 23108 hybrid_clock.cc:648] HybridClock initialized: now 1714593615778726 us; error 42 us; skew 500 ppm
May 01 20:00:15 dist-test-slave-tzct krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:15.781433 23108 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:00:15.782369 23108 webserver.cc:460] Webserver started at http://127.8.168.126:39257/ using document root <none> and password file <none>
I20240501 20:00:15.782897 23108 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:15.782958 23108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:15.786540 23108 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:00:15.788499 23123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:15.789541 23108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:15.789659 23108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "105abcddc47f43b3a2dcc204988e5c12"
format_stamp: "Formatted at 2024-05-01 20:00:13 on dist-test-slave-tzct"
server_key: "a5ed3b11e9c81c1afb542785a1e3c684"
server_key_iv: "67ef9404aad49452738f6c629dd3a27e"
server_key_version: "encryptionkey@0"
I20240501 20:00:15.790068 23108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:15.802239 23108 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:15.805016 23108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:15.805177 23108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:15.811566 23108 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:41909
I20240501 20:00:15.811585 23175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:41909 every 8 connection(s)
I20240501 20:00:15.812987 23108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:00:15.816933 23176 sys_catalog.cc:263] Verifying existing consensus state
I20240501 20:00:15.818480 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 23108
I20240501 20:00:15.818511 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12: Bootstrap starting.
I20240501 20:00:15.818848 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:44835
--local_ip_for_outbound_sockets=127.8.168.65
--tserver_master_addrs=127.8.168.126:41909
--webserver_port=43869
--webserver_interface=127.8.168.65
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 20:00:15.838645 23176 log.cc:824] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:15.851900 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:00:15.852337 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12: Bootstrap complete.
I20240501 20:00:15.856935 23176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } }
I20240501 20:00:15.857367 23176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105abcddc47f43b3a2dcc204988e5c12, State: Initialized, Role: FOLLOWER
I20240501 20:00:15.857900 23176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } }
I20240501 20:00:15.858042 23176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:00:15.858103 23176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:00:15.858206 23176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:15.860248 23176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } }
I20240501 20:00:15.860617 23176 leader_election.cc:302] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 105abcddc47f43b3a2dcc204988e5c12; no voters:
I20240501 20:00:15.860963 23176 leader_election.cc:288] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [CANDIDATE]: Term 3 election: Requested vote from peers
I20240501 20:00:15.861079 23180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:00:15.861316 23180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [term 3 LEADER]: Becoming Leader. State: Replica: 105abcddc47f43b3a2dcc204988e5c12, State: Running, Role: LEADER
I20240501 20:00:15.861644 23180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } }
I20240501 20:00:15.862076 23176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:00:15.863042 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 105abcddc47f43b3a2dcc204988e5c12. Latest consensus state: current_term: 3 leader_uuid: "105abcddc47f43b3a2dcc204988e5c12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } } }
I20240501 20:00:15.863191 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:15.863042 23181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "105abcddc47f43b3a2dcc204988e5c12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105abcddc47f43b3a2dcc204988e5c12" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 41909 } } }
I20240501 20:00:15.863453 23181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105abcddc47f43b3a2dcc204988e5c12 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:15.863721 23189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:00:15.868439 23189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:00:15.869235 23189 catalog_manager.cc:1259] Loaded cluster ID: 383f7000892044a9ae7c31e097a2321d
I20240501 20:00:15.869292 23189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20240501 20:00:15.870090 23189 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f60fb811dcd google::LogMessage::Fail() at ??:0
@ 0x7f60fb815b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f60fb8117cc google::LogMessage::Flush() at ??:0
@ 0x7f60fb812f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6100693308 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f61006941a0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f61006906a1 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f61006d7843 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55cabcdb075c std::function<>::operator()() at ??:0
@ 0x7f60fcc62e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f60fcc636ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f60fcc6503b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55cabcdb075c std::function<>::operator()() at ??:0
@ 0x7f60fcc52807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f60fd4906db start_thread at ??:0
@ 0x7f60fa9c871f clone at ??:0
W20240501 20:00:15.962472 23178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:15.962704 23178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:15.962760 23178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:15.965842 23178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:15.965901 23178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:15.965984 23178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 20:00:15.970002 23178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:44835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=43869
--tserver_master_addrs=127.8.168.126:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:15.970873 23178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:15.971673 23178 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:15.979904 23217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:15.980345 23178 server_base.cc:1034] running on GCE node
W20240501 20:00:15.980877 23220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:15.981282 23218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:15.981729 23178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:15.982349 23178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:15.983520 23178 hybrid_clock.cc:648] HybridClock initialized: now 1714593615983493 us; error 44 us; skew 500 ppm
May 01 20:00:15 dist-test-slave-tzct krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:15.986574 23178 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 20:00:15.987540 23178 webserver.cc:460] Webserver started at http://127.8.168.65:43869/ using document root <none> and password file <none>
I20240501 20:00:15.988058 23178 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:15.988114 23178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:15.991570 23178 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:15.993641 23227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:15.994904 23178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 20:00:15.995014 23178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "4ee8e00be6684bce8007800fcb7e9df5"
format_stamp: "Formatted at 2024-05-01 20:00:14 on dist-test-slave-tzct"
server_key: "584ed12a423040d313e424f0aac5be66"
server_key_iv: "142efdf649458d55ac5e978078b3f91d"
server_key_version: "encryptionkey@0"
I20240501 20:00:15.995446 23178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:16.011235 23178 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:16.014467 23178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:16.014644 23178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:16.015218 23178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:16.016319 23234 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 20:00:16.020419 23178 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 20:00:16.020488 23178 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20240501 20:00:16.020555 23178 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 20:00:16.022461 23178 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 20:00:16.022522 23178 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240501 20:00:16.022784 23234 tablet_bootstrap.cc:492] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5: Bootstrap starting.
I20240501 20:00:16.036974 23178 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:44835
I20240501 20:00:16.038471 23178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 20:00:16.038763 23341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:44835 every 8 connection(s)
I20240501 20:00:16.042758 23234 log.cc:824] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:16.045424 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 23178
I20240501 20:00:16.045866 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:41379
--local_ip_for_outbound_sockets=127.8.168.66
--tserver_master_addrs=127.8.168.126:41909
--webserver_port=45035
--webserver_interface=127.8.168.66
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 20:00:16.049322 23234 tablet_bootstrap.cc:492] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:00:16.049652 23234 tablet_bootstrap.cc:492] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5: Bootstrap complete.
I20240501 20:00:16.050621 23234 ts_tablet_manager.cc:1367] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5: Time spent bootstrapping tablet: real 0.028s user 0.017s sys 0.008s
I20240501 20:00:16.052647 23234 raft_consensus.cc:348] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } }
I20240501 20:00:16.053062 23234 raft_consensus.cc:729] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee8e00be6684bce8007800fcb7e9df5, State: Initialized, Role: FOLLOWER
I20240501 20:00:16.053570 23234 consensus_queue.cc:260] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } }
I20240501 20:00:16.054467 23234 ts_tablet_manager.cc:1398] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20240501 20:00:16.082059 23344 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:00:16.041259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:43793)
0501 20:00:16.041813 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:16.041865 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:16.042868 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:16.081331 (+ 38463us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":135,"threads_started":1}
W20240501 20:00:16.082547 23342 heartbeater.cc:644] Failed to heartbeat to 127.8.168.126:41909 (0 consecutive failures): Network error: Failed to ping master at 127.8.168.126:41909: Client connection negotiation failed: client connection to 127.8.168.126:41909: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:00:16.083245 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:16.082989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:33143)
0501 20:00:16.083118 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:16.083180 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20240501 20:00:16.083858 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:16.083653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:47129)
0501 20:00:16.083751 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:16.083795 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20240501 20:00:16.084045 23342 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240501 20:00:16.161257 23346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:16.161506 23346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:16.161562 23346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:16.164590 23346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:16.164649 23346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:16.164747 23346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 20:00:16.168773 23346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:41379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=45035
--tserver_master_addrs=127.8.168.126:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:16.169602 23346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:16.170405 23346 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:16.178565 23353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:16.178941 23355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:16.179065 23352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:16.179718 23346 server_base.cc:1034] running on GCE node
I20240501 20:00:16.180123 23346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:16.180701 23346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:16.181908 23346 hybrid_clock.cc:648] HybridClock initialized: now 1714593616181893 us; error 32 us; skew 500 ppm
May 01 20:00:16 dist-test-slave-tzct krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593616, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:16.189437 23346 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 20:00:16.190491 23346 webserver.cc:460] Webserver started at http://127.8.168.66:45035/ using document root <none> and password file <none>
I20240501 20:00:16.191076 23346 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:16.191145 23346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:16.194923 23346 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:16.196897 23362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:16.197934 23346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:00:16.198055 23346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00"
format_stamp: "Formatted at 2024-05-01 20:00:14 on dist-test-slave-tzct"
server_key: "45f1021b17a140f4110a34c9a7b00ee1"
server_key_iv: "5b0b7b3aff8d01ff900a87d453ba3646"
server_key_version: "encryptionkey@0"
I20240501 20:00:16.198522 23346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:16.216207 23346 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:16.219132 23346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:16.219316 23346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:16.219877 23346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:16.220692 23346 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:16.220788 23346 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:16.220844 23346 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:16.220870 23346 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:16.229266 23346 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:41379
I20240501 20:00:16.229295 23475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:41379 every 8 connection(s)
I20240501 20:00:16.230293 23346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 20:00:16.233436 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 23346
I20240501 20:00:16.233662 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:16.232163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:56323)
0501 20:00:16.232594 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:16.232697 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":91,"threads_started":1}
I20240501 20:00:16.234006 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:41511
--local_ip_for_outbound_sockets=127.8.168.67
--tserver_master_addrs=127.8.168.126:41909
--webserver_port=42549
--webserver_interface=127.8.168.67
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:16.234479 23476 heartbeater.cc:644] Failed to heartbeat to 127.8.168.126:41909 (0 consecutive failures): Network error: Failed to ping master at 127.8.168.126:41909: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
I20240501 20:00:16.235293 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:16.234985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:40487)
0501 20:00:16.235145 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:16.235204 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20240501 20:00:16.236018 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:16.235764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:42881)
0501 20:00:16.235899 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:16.235949 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20240501 20:00:16.236172 23476 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240501 20:00:16.343504 23479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:16.343742 23479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:16.343798 23479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:16.347247 23479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:16.347327 23479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:16.347421 23479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 20:00:16.351675 23479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:41511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=42549
--tserver_master_addrs=127.8.168.126:41909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:16.352584 23479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:16.353458 23479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:16.361652 23485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:16.361652 23484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:16.362160 23487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:16.362742 23479 server_base.cc:1034] running on GCE node
I20240501 20:00:16.363147 23479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:16.363759 23479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:16.364980 23479 hybrid_clock.cc:648] HybridClock initialized: now 1714593616364963 us; error 28 us; skew 500 ppm
May 01 20:00:16 dist-test-slave-tzct krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593616, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:16.367587 23479 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 20:00:16.368511 23479 webserver.cc:460] Webserver started at http://127.8.168.67:42549/ using document root <none> and password file <none>
I20240501 20:00:16.369083 23479 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:16.369153 23479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:16.372627 23479 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:16.374553 23494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:16.375553 23479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:16.375654 23479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "5d1cc43c44274fd18fddf7c91eca2806"
format_stamp: "Formatted at 2024-05-01 20:00:14 on dist-test-slave-tzct"
server_key: "8eb65fcff5dfa74d214964561f5ef21e"
server_key_iv: "b066f2595a33c03834f8cd1f8b4092d9"
server_key_version: "encryptionkey@0"
I20240501 20:00:16.376047 23479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:16.389853 23479 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:16.392387 23479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:16.392541 23479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:16.393083 23479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:16.393853 23479 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:16.393898 23479 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:16.393949 23479 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:16.393978 23479 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:16.402271 23479 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:41511
I20240501 20:00:16.402326 23607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:41511 every 8 connection(s)
I20240501 20:00:16.403339 23479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 20:00:16.406440 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:16.405152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:40945)
0501 20:00:16.405592 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:16.405682 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":146,"threads_started":1}
W20240501 20:00:16.406893 23608 heartbeater.cc:644] Failed to heartbeat to 127.8.168.126:41909 (0 consecutive failures): Network error: Failed to ping master at 127.8.168.126:41909: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
I20240501 20:00:16.407420 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:16.407234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:40021)
0501 20:00:16.407327 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:16.407367 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20240501 20:00:16.407935 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:16.407747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:55513)
0501 20:00:16.407834 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:16.407888 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20240501 20:00:16.408066 23608 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240501 20:00:16.410666 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 23479
I20240501 20:00:17.085024 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:17.084586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:59703)
0501 20:00:17.084858 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:17.084912 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":87,"threads_started":1}
I20240501 20:00:17.237272 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:17.236836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:32969)
0501 20:00:17.237088 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:17.237148 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20240501 20:00:17.409029 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:17.408612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:50537)
0501 20:00:17.408872 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:17.408925 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":93,"threads_started":1}
I20240501 20:00:17.693143 23615 raft_consensus.cc:482] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:17.693339 23615 raft_consensus.cc:504] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } }
I20240501 20:00:17.694342 23615 leader_election.cc:288] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379), 5d1cc43c44274fd18fddf7c91eca2806 (127.8.168.67:41511)
May 01 20:00:17 dist-test-slave-tzct krb5kdc[22034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.67@KRBTEST.COM
May 01 20:00:17 dist-test-slave-tzct krb5kdc[22034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593615, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.66@KRBTEST.COM
I20240501 20:00:17.705777 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:17.694597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:41379 (local address 127.8.168.65:49771)
0501 20:00:17.695105 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:17.695120 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:17.695242 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:17.696592 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:17.696598 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:17.697187 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:17.697675 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:17.697688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:17.698695 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:17.698698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:17.699071 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:17.699078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:17.699216 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:17.699860 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:17.699884 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:17.701955 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:17.703890 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:17.703893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:17.703895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:17.704120 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:17.704374 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:17.704377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:17.704379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:17.704454 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:17.704989 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:17.704994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:17.705188 (+ 194us) client_negotiation.cc:769] Sending connection context
0501 20:00:17.705353 (+ 165us) client_negotiation.cc:240] Negotiation successful
0501 20:00:17.705597 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"spinlock_wait_cycles":7936,"thread_start_us":83,"threads_started":1}
I20240501 20:00:17.705842 23617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:17.694760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:49771 (local address 127.8.168.66:41379)
0501 20:00:17.695043 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:17.695055 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:17.695227 (+ 172us) server_negotiation.cc:407] Connection header received
0501 20:00:17.695968 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:17.695974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:17.696249 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:17.696539 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:17.697826 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:17.698571 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:17.699176 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:17.699425 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:17.702078 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:17.702114 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:17.702122 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:17.702153 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:17.703784 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:17.704219 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:17.704223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:17.704227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:17.704292 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:17.704534 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:17.704537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:17.704539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:17.704885 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:17.705009 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:17.705479 (+ 470us) server_negotiation.cc:299] Negotiation successful
0501 20:00:17.705691 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":81,"threads_started":1}
I20240501 20:00:17.705868 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:17.695135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:52135 (local address 127.8.168.67:41511)
0501 20:00:17.695424 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:17.695432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:17.695460 (+ 28us) server_negotiation.cc:407] Connection header received
0501 20:00:17.695996 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:17.696002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:17.696250 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:17.696539 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:17.697828 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:17.698566 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:17.699163 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:17.699450 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:17.701876 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:17.701917 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:17.701927 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:17.701958 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:17.703690 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:17.704215 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:17.704219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:17.704223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:17.704290 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:17.704535 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:17.704538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:17.704540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:17.704885 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:17.705009 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:17.705479 (+ 470us) server_negotiation.cc:299] Negotiation successful
0501 20:00:17.705688 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20240501 20:00:17.706398 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:17.694975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:41511 (local address 127.8.168.65:52135)
0501 20:00:17.695223 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:17.695236 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:17.695333 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:17.696592 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:17.696598 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:17.697186 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:17.697675 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:17.697688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:17.698709 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:17.698712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:17.699065 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:17.699072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:17.699212 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:17.699914 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:17.699930 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:17.701707 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:17.703798 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:17.703804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:17.703813 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:17.704112 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:17.704391 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:17.704394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:17.704396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:17.704457 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:17.704988 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:17.704994 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:17.705188 (+ 194us) client_negotiation.cc:769] Sending connection context
0501 20:00:17.705353 (+ 165us) client_negotiation.cc:240] Negotiation successful
0501 20:00:17.705689 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":115,"spinlock_wait_cycles":32640,"thread_start_us":66,"threads_started":1}
W20240501 20:00:17.707911 23231 leader_election.cc:341] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379): Not found: Tablet not found: 38a614223ae24bb5ade4c22160fb9d5e
W20240501 20:00:17.708092 23231 leader_election.cc:341] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d1cc43c44274fd18fddf7c91eca2806 (127.8.168.67:41511): Not found: Tablet not found: 38a614223ae24bb5ade4c22160fb9d5e
I20240501 20:00:17.708153 23231 leader_election.cc:302] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ee8e00be6684bce8007800fcb7e9df5; no voters: 5d1cc43c44274fd18fddf7c91eca2806, b6f662d52dbc4fd8bf0fdcc9aaa44c00
I20240501 20:00:17.708390 23615 raft_consensus.cc:2726] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:00:18.086102 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:18.085791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:38931)
0501 20:00:18.085946 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:18.086007 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20240501 20:00:18.238438 23620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:18.237991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:42643)
0501 20:00:18.238267 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:18.238325 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
I20240501 20:00:18.410094 23621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:18.409721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:58295)
0501 20:00:18.409951 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:18.410003 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
I20240501 20:00:19.087337 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:19.086896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:38711)
0501 20:00:19.087174 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:19.087232 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20240501 20:00:19.240236 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:19.239737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:39071)
0501 20:00:19.240040 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:19.240106 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
I20240501 20:00:19.411265 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:19.410829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:49769)
0501 20:00:19.411104 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:19.411164 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
I20240501 20:00:19.525540 23625 raft_consensus.cc:482] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:19.525674 23625 raft_consensus.cc:504] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } }
I20240501 20:00:19.526085 23625 leader_election.cc:288] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379), 5d1cc43c44274fd18fddf7c91eca2806 (127.8.168.67:41511)
W20240501 20:00:19.526964 23231 leader_election.cc:341] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379): Not found: Tablet not found: 38a614223ae24bb5ade4c22160fb9d5e
W20240501 20:00:19.527096 23231 leader_election.cc:341] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d1cc43c44274fd18fddf7c91eca2806 (127.8.168.67:41511): Not found: Tablet not found: 38a614223ae24bb5ade4c22160fb9d5e
I20240501 20:00:19.527127 23231 leader_election.cc:302] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ee8e00be6684bce8007800fcb7e9df5; no voters: 5d1cc43c44274fd18fddf7c91eca2806, b6f662d52dbc4fd8bf0fdcc9aaa44c00
I20240501 20:00:19.527269 23625 raft_consensus.cc:2726] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:00:20.088515 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:20.088089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:53735)
0501 20:00:20.088347 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:20.088412 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":91,"threads_started":1}
I20240501 20:00:20.241524 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:20.241086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:32895)
0501 20:00:20.241352 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:20.241420 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20240501 20:00:20.412461 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:20.412020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:45075)
0501 20:00:20.412293 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:20.412345 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
I20240501 20:00:21.089797 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:21.089404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:35547)
0501 20:00:21.089648 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:21.089699 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
I20240501 20:00:21.242640 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:21.242226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:49799)
0501 20:00:21.242491 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:21.242544 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":103,"threads_started":1}
I20240501 20:00:21.413558 23631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:21.413141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:33437)
0501 20:00:21.413401 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:21.413457 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":84,"threads_started":1}
I20240501 20:00:21.580602 23632 raft_consensus.cc:482] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:21.580749 23632 raft_consensus.cc:504] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } }
I20240501 20:00:21.581164 23632 leader_election.cc:288] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379), 5d1cc43c44274fd18fddf7c91eca2806 (127.8.168.67:41511)
W20240501 20:00:21.582062 23231 leader_election.cc:341] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d1cc43c44274fd18fddf7c91eca2806 (127.8.168.67:41511): Not found: Tablet not found: 38a614223ae24bb5ade4c22160fb9d5e
W20240501 20:00:21.582178 23231 leader_election.cc:341] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379): Not found: Tablet not found: 38a614223ae24bb5ade4c22160fb9d5e
I20240501 20:00:21.582221 23231 leader_election.cc:302] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ee8e00be6684bce8007800fcb7e9df5; no voters: 5d1cc43c44274fd18fddf7c91eca2806, b6f662d52dbc4fd8bf0fdcc9aaa44c00
I20240501 20:00:21.582376 23632 raft_consensus.cc:2726] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:00:22.090981 23633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:22.090533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:50231)
0501 20:00:22.090804 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:22.090868 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20240501 20:00:22.243692 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:22.243311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:49419)
0501 20:00:22.243548 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:22.243596 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
I20240501 20:00:22.414697 23635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:22.414276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:52425)
0501 20:00:22.414550 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:22.414604 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":111,"threads_started":1}
I20240501 20:00:23.092175 23636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:23.091741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:37135)
0501 20:00:23.092020 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:23.092070 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":98,"threads_started":1}
I20240501 20:00:23.244825 23637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:23.244395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:55201)
0501 20:00:23.244650 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:23.244700 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":72,"threads_started":1}
I20240501 20:00:23.415884 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:23.415433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:51121)
0501 20:00:23.415724 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:23.415786 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":86,"threads_started":1}
I20240501 20:00:24.093411 23639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:24.092966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:51517)
0501 20:00:24.093215 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:24.093270 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":83,"threads_started":1}
I20240501 20:00:24.187342 23640 raft_consensus.cc:482] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:24.187464 23640 raft_consensus.cc:504] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } }
I20240501 20:00:24.187883 23640 leader_election.cc:288] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379), 5d1cc43c44274fd18fddf7c91eca2806 (127.8.168.67:41511)
W20240501 20:00:24.188755 23231 leader_election.cc:341] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d1cc43c44274fd18fddf7c91eca2806 (127.8.168.67:41511): Not found: Tablet not found: 38a614223ae24bb5ade4c22160fb9d5e
W20240501 20:00:24.188894 23231 leader_election.cc:341] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379): Not found: Tablet not found: 38a614223ae24bb5ade4c22160fb9d5e
I20240501 20:00:24.188947 23231 leader_election.cc:302] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ee8e00be6684bce8007800fcb7e9df5; no voters: 5d1cc43c44274fd18fddf7c91eca2806, b6f662d52dbc4fd8bf0fdcc9aaa44c00
I20240501 20:00:24.189119 23640 raft_consensus.cc:2726] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:00:24.245868 23641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:24.245491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:59027)
0501 20:00:24.245719 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:24.245769 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":72,"threads_started":1}
I20240501 20:00:24.417201 23642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:24.416700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:55189)
0501 20:00:24.416995 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:24.417059 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
I20240501 20:00:25.094630 23643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:25.094186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:45179)
0501 20:00:25.094459 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:25.094525 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
I20240501 20:00:25.246948 23644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:25.246557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:59129)
0501 20:00:25.246791 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:25.246845 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20240501 20:00:25.418511 23645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:25.417930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:36497)
0501 20:00:25.418307 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:25.418407 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":97,"threads_started":1}
I20240501 20:00:26.095933 23646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:26.095467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:33697)
0501 20:00:26.095755 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:26.095812 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":78,"threads_started":1}
I20240501 20:00:26.248001 23647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:26.247616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:42225)
0501 20:00:26.247849 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:26.247897 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
I20240501 20:00:26.419678 23648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:26.419277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:42543)
0501 20:00:26.419526 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:26.419577 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":79,"threads_started":1}
I20240501 20:00:27.097062 23649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:27.096655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:48755)
0501 20:00:27.096911 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:27.096962 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
I20240501 20:00:27.249243 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:27.248755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:60335)
0501 20:00:27.249063 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:27.249129 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":97,"threads_started":1}
I20240501 20:00:27.420900 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:27.420446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:42551)
0501 20:00:27.420706 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:27.420797 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
I20240501 20:00:28.098276 23652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:28.097864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:50967)
0501 20:00:28.098123 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:28.098176 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":92,"threads_started":1}
I20240501 20:00:28.250454 23653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:28.250010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:60995)
0501 20:00:28.250291 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:28.250351 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
I20240501 20:00:28.422116 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:28.421647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:59699)
0501 20:00:28.421953 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:28.422014 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":102,"threads_started":1}
I20240501 20:00:29.099414 23655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:29.098996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:52009)
0501 20:00:29.099229 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:29.099279 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
I20240501 20:00:29.251600 23656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:29.251183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:47677)
0501 20:00:29.251446 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:29.251498 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20240501 20:00:29.423288 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:29.422900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:36447)
0501 20:00:29.423141 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:29.423189 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":89,"threads_started":1}
I20240501 20:00:30.100503 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:30.100127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:42091)
0501 20:00:30.100358 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:30.100406 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":74,"threads_started":1}
I20240501 20:00:30.252658 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:30.252281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:43485)
0501 20:00:30.252512 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:30.252560 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":74,"threads_started":1}
I20240501 20:00:30.350654 23660 raft_consensus.cc:482] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:30.350780 23660 raft_consensus.cc:504] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6f662d52dbc4fd8bf0fdcc9aaa44c00" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 41379 } } peers { permanent_uuid: "5d1cc43c44274fd18fddf7c91eca2806" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 41511 } } peers { permanent_uuid: "4ee8e00be6684bce8007800fcb7e9df5" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 44835 } }
I20240501 20:00:30.351164 23660 leader_election.cc:288] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379), 5d1cc43c44274fd18fddf7c91eca2806 (127.8.168.67:41511)
W20240501 20:00:30.351980 23231 leader_election.cc:341] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6f662d52dbc4fd8bf0fdcc9aaa44c00 (127.8.168.66:41379): Not found: Tablet not found: 38a614223ae24bb5ade4c22160fb9d5e
W20240501 20:00:30.352114 23231 leader_election.cc:341] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d1cc43c44274fd18fddf7c91eca2806 (127.8.168.67:41511): Not found: Tablet not found: 38a614223ae24bb5ade4c22160fb9d5e
I20240501 20:00:30.352164 23231 leader_election.cc:302] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ee8e00be6684bce8007800fcb7e9df5; no voters: 5d1cc43c44274fd18fddf7c91eca2806, b6f662d52dbc4fd8bf0fdcc9aaa44c00
I20240501 20:00:30.352326 23660 raft_consensus.cc:2726] T 38a614223ae24bb5ade4c22160fb9d5e P 4ee8e00be6684bce8007800fcb7e9df5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:00:30.424432 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:30.424014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:51463)
0501 20:00:30.424284 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:30.424334 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":91,"threads_started":1}
I20240501 20:00:31.101799 23662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:31.101354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.65:36075)
0501 20:00:31.101636 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:31.101693 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20240501 20:00:31.253938 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:31.253504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.66:56489)
0501 20:00:31.253782 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:31.253839 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":76,"threads_started":1}
I20240501 20:00:31.420397 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 23178
I20240501 20:00:31.425251 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 23346
I20240501 20:00:31.426237 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:31.425271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:41909 (local address 127.8.168.67:57183)
0501 20:00:31.426063 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:31.426123 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":66,"threads_started":1}
I20240501 20:00:31.429740 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 23479
2024-05-01T20:00:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18734 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:00:31 dist-test-slave-tzct krb5kdc[23667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:00:31 dist-test-slave-tzct krb5kdc[23667](info): set up 2 sockets
May 01 20:00:31 dist-test-slave-tzct krb5kdc[23667](info): commencing operation
krb5kdc: starting...
W20240501 20:00:32.317138 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.867s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:00:32 dist-test-slave-tzct krb5kdc[23667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:00:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:00:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 20:00:32.429853 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:39523
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:39523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:32.535193 23680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:32.535465 23680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:32.535537 23680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:32.538573 23680 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20240501 20:00:32.538625 23680 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20240501 20:00:32.538647 23680 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20240501 20:00:32.538673 23680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:00:32.538700 23680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:32.538738 23680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:32.538769 23680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:00:32.538800 23680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:00:32.542852 23680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:39523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:39523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:32.543751 23680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:32.544623 23680 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:32.552304 23688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:32.552834 23680 server_base.cc:1034] running on GCE node
W20240501 20:00:32.552881 23685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:32.553301 23686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:32.553802 23680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:32.554718 23680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:32.555869 23680 hybrid_clock.cc:648] HybridClock initialized: now 1714593632555847 us; error 37 us; skew 500 ppm
May 01 20:00:32 dist-test-slave-tzct krb5kdc[23667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593632, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:32.558487 23680 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:00:32.559407 23680 webserver.cc:460] Webserver started at http://127.8.168.126:41413/ using document root <none> and password file <none>
I20240501 20:00:32.559913 23680 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:32.559963 23680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:32.560122 23680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:32.561902 23680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "d43c9a64005e41efab8421ad002ddf76"
format_stamp: "Formatted at 2024-05-01 20:00:32 on dist-test-slave-tzct"
server_key: "6151a5281cdbf86d42ce39f35731b5ec"
server_key_iv: "e8baaa7c478998be38e53b28cf4612d3"
server_key_version: "encryptionkey@0"
I20240501 20:00:32.562327 23680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "d43c9a64005e41efab8421ad002ddf76"
format_stamp: "Formatted at 2024-05-01 20:00:32 on dist-test-slave-tzct"
server_key: "6151a5281cdbf86d42ce39f35731b5ec"
server_key_iv: "e8baaa7c478998be38e53b28cf4612d3"
server_key_version: "encryptionkey@0"
I20240501 20:00:32.565487 23680 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:00:32.567566 23695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:32.568507 23680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:00:32.568620 23680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "d43c9a64005e41efab8421ad002ddf76"
format_stamp: "Formatted at 2024-05-01 20:00:32 on dist-test-slave-tzct"
server_key: "6151a5281cdbf86d42ce39f35731b5ec"
server_key_iv: "e8baaa7c478998be38e53b28cf4612d3"
server_key_version: "encryptionkey@0"
I20240501 20:00:32.568702 23680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:32.594918 23680 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:32.597513 23680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:32.597662 23680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:32.604034 23680 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:39523
I20240501 20:00:32.604256 23747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:39523 every 8 connection(s)
I20240501 20:00:32.605032 23680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:00:32.607429 23748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:32.612201 23748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: Bootstrap starting.
I20240501 20:00:32.614446 23748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:32.615144 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 23680
I20240501 20:00:32.615137 23748 log.cc:824] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:32.615273 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:00:32.615546 8865 external_mini_cluster.cc:1428] Setting key 4b7b8f0236f1d24768e413d97d1b9fc6
I20240501 20:00:32.617007 23748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: No bootstrap required, opened a new log
I20240501 20:00:32.619325 23748 raft_consensus.cc:348] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } }
I20240501 20:00:32.619491 23748 raft_consensus.cc:374] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:32.619524 23748 raft_consensus.cc:729] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43c9a64005e41efab8421ad002ddf76, State: Initialized, Role: FOLLOWER
I20240501 20:00:32.619905 23748 consensus_queue.cc:260] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } }
I20240501 20:00:32.620023 23748 raft_consensus.cc:388] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:00:32.620080 23748 raft_consensus.cc:482] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:00:32.620184 23748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:32.620954 23748 raft_consensus.cc:504] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } }
I20240501 20:00:32.621294 23748 leader_election.cc:302] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43c9a64005e41efab8421ad002ddf76; no voters:
I20240501 20:00:32.621594 23748 leader_election.cc:288] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:00:32.621676 23753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:32.621850 23753 raft_consensus.cc:686] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 1 LEADER]: Becoming Leader. State: Replica: d43c9a64005e41efab8421ad002ddf76, State: Running, Role: LEADER
I20240501 20:00:32.622191 23753 consensus_queue.cc:237] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } }
I20240501 20:00:32.622709 23748 sys_catalog.cc:564] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [sys.catalog]: configured and running, proceeding with master startup.
May 01 20:00:32 dist-test-slave-tzct krb5kdc[23667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:32.623651 23755 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d43c9a64005e41efab8421ad002ddf76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } } }
I20240501 20:00:32.623765 23755 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:32.624027 23754 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d43c9a64005e41efab8421ad002ddf76. Latest consensus state: current_term: 1 leader_uuid: "d43c9a64005e41efab8421ad002ddf76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } } }
I20240501 20:00:32.624193 23754 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:32.624382 23762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:00:32.627715 23762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:00:32.629384 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:32.616568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52794 (local address 127.8.168.126:39523)
0501 20:00:32.616964 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:32.616973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:32.617007 (+ 34us) server_negotiation.cc:407] Connection header received
0501 20:00:32.617619 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:32.617639 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:32.617910 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:32.618219 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:32.619090 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:32.620286 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:32.620969 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:32.621291 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:32.623712 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:32.623732 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:32.623745 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:32.623788 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:32.626261 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:32.626724 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:32.626729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:32.626734 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:32.626824 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:32.627061 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:32.627064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:32.627065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:32.627486 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:32.627694 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:32.628344 (+ 650us) server_negotiation.cc:299] Negotiation successful
0501 20:00:32.628617 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":123,"threads_started":1}
I20240501 20:00:32.632858 23762 catalog_manager.cc:1347] Generated new cluster ID: c53a43c37cb146a5a2f0ad2dd9359472
I20240501 20:00:32.632922 23762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:00:32.650013 23762 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:00:32.650978 23762 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:00:32.661046 23762 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: Generated new TSK 0
I20240501 20:00:32.661717 23762 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
I20240501 20:00:32.697649 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:39523
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:32.804265 23774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:32.804508 23774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:32.804569 23774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:32.807590 23774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:32.807648 23774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:32.807740 23774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 20:00:32.811717 23774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:39523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:32.812572 23774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:32.813390 23774 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:32.821348 23779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:32.821645 23774 server_base.cc:1034] running on GCE node
W20240501 20:00:32.822057 23782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:32.822422 23780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:32.822825 23774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:32.823403 23774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:32.824559 23774 hybrid_clock.cc:648] HybridClock initialized: now 1714593632824530 us; error 42 us; skew 500 ppm
May 01 20:00:32 dist-test-slave-tzct krb5kdc[23667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593632, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:32.827097 23774 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 20:00:32.827937 23774 webserver.cc:460] Webserver started at http://127.8.168.65:35677/ using document root <none> and password file <none>
I20240501 20:00:32.828444 23774 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:32.828505 23774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:32.828667 23774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:32.830350 23774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "b8ba60fb865a42c8bdf5b26b2e647259"
format_stamp: "Formatted at 2024-05-01 20:00:32 on dist-test-slave-tzct"
server_key: "5886a6aaaffc4a84ad0c7c9bd276ac9a"
server_key_iv: "e9c39f5e79127df8c784460527f53ff1"
server_key_version: "encryptionkey@0"
I20240501 20:00:32.830802 23774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "b8ba60fb865a42c8bdf5b26b2e647259"
format_stamp: "Formatted at 2024-05-01 20:00:32 on dist-test-slave-tzct"
server_key: "5886a6aaaffc4a84ad0c7c9bd276ac9a"
server_key_iv: "e9c39f5e79127df8c784460527f53ff1"
server_key_version: "encryptionkey@0"
I20240501 20:00:32.833874 23774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:00:32.835861 23789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:32.836855 23774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:32.836951 23774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "b8ba60fb865a42c8bdf5b26b2e647259"
format_stamp: "Formatted at 2024-05-01 20:00:32 on dist-test-slave-tzct"
server_key: "5886a6aaaffc4a84ad0c7c9bd276ac9a"
server_key_iv: "e9c39f5e79127df8c784460527f53ff1"
server_key_version: "encryptionkey@0"
I20240501 20:00:32.837030 23774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:32.857102 23774 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:32.859989 23774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:32.860148 23774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:32.860656 23774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:32.861438 23774 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:32.861496 23774 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:32.861549 23774 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:32.861577 23774 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:32.869423 23774 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:35715
I20240501 20:00:32.869462 23902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:35715 every 8 connection(s)
I20240501 20:00:32.870318 23774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 20:00:32.873229 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 23774
I20240501 20:00:32.873360 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 20:00:32.873668 8865 external_mini_cluster.cc:1428] Setting key 72ac8c8085d660ae872656b1f85c86b0
May 01 20:00:32 dist-test-slave-tzct krb5kdc[23667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593632, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.168.66@KRBTEST.COM; defaulting to no policy
I20240501 20:00:32.883003 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:32.872023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:51075 (local address 127.8.168.126:39523)
0501 20:00:32.872165 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:32.872169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:32.872845 (+ 676us) server_negotiation.cc:407] Connection header received
0501 20:00:32.874297 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:32.874300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:32.874355 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:32.874430 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:32.875996 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:32.876502 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:32.877217 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:32.877354 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:32.879635 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:32.879656 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:32.879660 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:32.879690 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:32.881214 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:32.881666 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:32.881669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:32.881671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:32.881721 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:32.881963 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:32.881966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:32.881968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:32.882168 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:32.882278 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:32.882741 (+ 463us) server_negotiation.cc:299] Negotiation successful
0501 20:00:32.882876 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 20:00:32.883638 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:32.872276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:51075)
0501 20:00:32.872666 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:32.872701 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:32.874087 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:32.874608 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:32.874620 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:32.875076 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:32.875836 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:32.875855 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:32.876627 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:32.876631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:32.877112 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:32.877120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:32.877289 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:32.877865 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:32.877889 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:32.879467 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:32.881325 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:32.881329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:32.881338 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:32.881565 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:32.881806 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:32.881808 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:32.881809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:32.881886 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:32.882261 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:32.882265 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:32.882492 (+ 227us) client_negotiation.cc:769] Sending connection context
0501 20:00:32.882678 (+ 186us) client_negotiation.cc:240] Negotiation successful
0501 20:00:32.882909 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":95,"threads_started":1}
I20240501 20:00:32.884809 23903 heartbeater.cc:347] Connected to a master server at 127.8.168.126:39523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.66@KRBTEST.COM" created.
I20240501 20:00:32.885041 23903 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:32.885649 23903 heartbeater.cc:511] Master 127.8.168.126:39523 requested a full tablet report, sending...
I20240501 20:00:32.887118 23712 ts_manager.cc:194] Registered new tserver with Master: b8ba60fb865a42c8bdf5b26b2e647259 (127.8.168.65:35715)
I20240501 20:00:32.888620 23712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:51075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
I20240501 20:00:32.900333 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:39523
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:33.003263 23908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:33.003504 23908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:33.003561 23908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:33.006589 23908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:33.006647 23908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:33.006722 23908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 20:00:33.010710 23908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:39523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:33.011528 23908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:33.012296 23908 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:33.020143 23916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:33.020537 23914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:33.021047 23913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:33.021409 23908 server_base.cc:1034] running on GCE node
I20240501 20:00:33.021775 23908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:33.022302 23908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:33.023442 23908 hybrid_clock.cc:648] HybridClock initialized: now 1714593633023383 us; error 70 us; skew 500 ppm
May 01 20:00:33 dist-test-slave-tzct krb5kdc[23667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593633, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:33.026093 23908 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 20:00:33.026995 23908 webserver.cc:460] Webserver started at http://127.8.168.66:37473/ using document root <none> and password file <none>
I20240501 20:00:33.027491 23908 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:33.027539 23908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:33.027683 23908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:33.029428 23908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "a823d5a9c1134ecfae1bed89bce5c697"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-tzct"
server_key: "5342e2f07c3fc12fc5f8584fe31b9d40"
server_key_iv: "16c93e65cb43f98f3178d77bdcb7cdc6"
server_key_version: "encryptionkey@0"
I20240501 20:00:33.029856 23908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "a823d5a9c1134ecfae1bed89bce5c697"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-tzct"
server_key: "5342e2f07c3fc12fc5f8584fe31b9d40"
server_key_iv: "16c93e65cb43f98f3178d77bdcb7cdc6"
server_key_version: "encryptionkey@0"
I20240501 20:00:33.032999 23908 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:00:33.035094 23923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:33.036057 23908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:33.036149 23908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "a823d5a9c1134ecfae1bed89bce5c697"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-tzct"
server_key: "5342e2f07c3fc12fc5f8584fe31b9d40"
server_key_iv: "16c93e65cb43f98f3178d77bdcb7cdc6"
server_key_version: "encryptionkey@0"
I20240501 20:00:33.036223 23908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:33.054558 23908 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:33.057261 23908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:33.057399 23908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:33.057881 23908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:33.058615 23908 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:33.058660 23908 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:33.058698 23908 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:33.058722 23908 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:33.067144 23908 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:37011
I20240501 20:00:33.067246 24036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:37011 every 8 connection(s)
I20240501 20:00:33.068049 23908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 20:00:33.075654 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 23908
I20240501 20:00:33.075790 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 20:00:33.076074 8865 external_mini_cluster.cc:1428] Setting key 7968c8da5615eb05efd27265c931b76a
May 01 20:00:33 dist-test-slave-tzct krb5kdc[23667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593633, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:33.080391 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:33.069703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:46335 (local address 127.8.168.126:39523)
0501 20:00:33.069829 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:33.069832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:33.070501 (+ 669us) server_negotiation.cc:407] Connection header received
0501 20:00:33.071626 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:33.071629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:33.071674 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:33.071745 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:33.073110 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.073646 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:33.074356 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.074512 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:33.076858 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:33.076873 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:33.076876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:33.076900 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:33.078419 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:33.078942 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:33.078946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:33.078948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:33.078998 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:33.079269 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:33.079272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:33.079274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:33.079454 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:33.079556 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:33.080155 (+ 599us) server_negotiation.cc:299] Negotiation successful
0501 20:00:33.080266 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240501 20:00:33.081300 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:33.069959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:46335)
0501 20:00:33.070340 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:33.070373 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:33.071450 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:33.071908 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:33.071915 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:33.072329 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:33.072983 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:33.072997 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.073791 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:33.073794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:33.074230 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:33.074239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.074440 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:33.075027 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:33.075053 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:33.076681 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:33.078539 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:33.078546 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:33.078557 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:33.078809 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:33.079084 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:33.079088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:33.079090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:33.079183 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:33.079561 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:33.079568 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:33.079882 (+ 314us) client_negotiation.cc:769] Sending connection context
0501 20:00:33.080121 (+ 239us) client_negotiation.cc:240] Negotiation successful
0501 20:00:33.080412 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":88,"threads_started":1}
I20240501 20:00:33.082345 24037 heartbeater.cc:347] Connected to a master server at 127.8.168.126:39523
I20240501 20:00:33.082609 24037 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:33.083259 24037 heartbeater.cc:511] Master 127.8.168.126:39523 requested a full tablet report, sending...
I20240501 20:00:33.084292 23712 ts_manager.cc:194] Registered new tserver with Master: a823d5a9c1134ecfae1bed89bce5c697 (127.8.168.66:37011)
I20240501 20:00:33.084861 23712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:46335
WARNING: no policy specified for kudu/127.8.168.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
I20240501 20:00:33.103494 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:39523
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:33.207579 24042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:33.207835 24042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:33.207901 24042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:33.210961 24042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:33.211028 24042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:33.211117 24042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 20:00:33.215274 24042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:39523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:33.216127 24042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:33.216966 24042 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:33.224916 24047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:33.225066 24048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:33.225389 24050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:33.225941 24042 server_base.cc:1034] running on GCE node
I20240501 20:00:33.226300 24042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:33.226819 24042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:33.227970 24042 hybrid_clock.cc:648] HybridClock initialized: now 1714593633227950 us; error 32 us; skew 500 ppm
May 01 20:00:33 dist-test-slave-tzct krb5kdc[23667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593633, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:33.230568 24042 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 20:00:33.231400 24042 webserver.cc:460] Webserver started at http://127.8.168.67:41535/ using document root <none> and password file <none>
I20240501 20:00:33.231905 24042 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:33.231961 24042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:33.232120 24042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:33.233863 24042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "5aa7730f481b4205b2813adf412b5572"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-tzct"
server_key: "465d77d6537b7b7d8f82505574f85fb8"
server_key_iv: "5462dfd0b99d45605b53340821222c1a"
server_key_version: "encryptionkey@0"
I20240501 20:00:33.234323 24042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "5aa7730f481b4205b2813adf412b5572"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-tzct"
server_key: "465d77d6537b7b7d8f82505574f85fb8"
server_key_iv: "5462dfd0b99d45605b53340821222c1a"
server_key_version: "encryptionkey@0"
I20240501 20:00:33.237428 24042 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:00:33.239462 24057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:33.240499 24042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:33.240605 24042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "5aa7730f481b4205b2813adf412b5572"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-tzct"
server_key: "465d77d6537b7b7d8f82505574f85fb8"
server_key_iv: "5462dfd0b99d45605b53340821222c1a"
server_key_version: "encryptionkey@0"
I20240501 20:00:33.240687 24042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:33.251334 24042 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:33.254184 24042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:33.254338 24042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:33.254842 24042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:33.255574 24042 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:33.255622 24042 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:33.255671 24042 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:33.255697 24042 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:33.263381 24042 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:34983
I20240501 20:00:33.263411 24170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:34983 every 8 connection(s)
I20240501 20:00:33.264251 24042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 20:00:33.268752 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 24042
I20240501 20:00:33.268868 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 20:00:33.269119 8865 external_mini_cluster.cc:1428] Setting key 6c775dfc79515157a5a87a7f5ed27592
May 01 20:00:33 dist-test-slave-tzct krb5kdc[23667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593633, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:33.276348 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:33.265878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:55551 (local address 127.8.168.126:39523)
0501 20:00:33.266028 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:33.266031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:33.266644 (+ 613us) server_negotiation.cc:407] Connection header received
0501 20:00:33.267814 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:33.267818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:33.267877 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:33.267950 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:33.269454 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.270082 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:33.270796 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.270933 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:33.273193 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:33.273210 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:33.273212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:33.273236 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:33.274577 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:33.274982 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:33.274984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:33.274985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:33.275024 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:33.275266 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:33.275269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:33.275270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:33.275475 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:33.275570 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:33.276117 (+ 547us) server_negotiation.cc:299] Negotiation successful
0501 20:00:33.276225 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20240501 20:00:33.276973 24173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:33.266125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:55551)
0501 20:00:33.266515 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:33.266548 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:33.267611 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:33.268122 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:33.268130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:33.268495 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:33.269295 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:33.269314 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.270228 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:33.270231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:33.270700 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:33.270709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.270917 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:33.271517 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:33.271541 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:33.272984 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:33.274674 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:33.274678 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:33.274687 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:33.274907 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:33.275089 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:33.275091 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:33.275092 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:33.275166 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:33.275611 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:33.275616 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:33.275850 (+ 234us) client_negotiation.cc:769] Sending connection context
0501 20:00:33.276037 (+ 187us) client_negotiation.cc:240] Negotiation successful
0501 20:00:33.276259 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":92,"threads_started":1}
I20240501 20:00:33.278121 24171 heartbeater.cc:347] Connected to a master server at 127.8.168.126:39523
I20240501 20:00:33.278326 24171 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:33.278796 24171 heartbeater.cc:511] Master 127.8.168.126:39523 requested a full tablet report, sending...
I20240501 20:00:33.279817 23712 ts_manager.cc:194] Registered new tserver with Master: 5aa7730f481b4205b2813adf412b5572 (127.8.168.67:34983)
I20240501 20:00:33.280340 23712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:55551
I20240501 20:00:33.282829 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 20:00:33.290555 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:33.284556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.8.168.126:39523)
0501 20:00:33.284681 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:33.284684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:33.284836 (+ 152us) server_negotiation.cc:407] Connection header received
0501 20:00:33.284974 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:33.284977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:33.285024 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:33.285103 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:33.285821 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.286328 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:33.286923 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.287081 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:33.288088 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:33.288102 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:33.288104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:33.288127 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:33.289489 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:33.289788 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:33.289790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:33.289792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:33.289827 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:33.289966 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:33.289968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:33.289969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:33.290115 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:33.290209 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:33.290343 (+ 134us) server_negotiation.cc:299] Negotiation successful
0501 20:00:33.290436 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240501 20:00:33.293187 23712 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:00:33.295615 23712 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:00:33.307348 24183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:33.303243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:37011 (local address 127.0.0.1:42286)
0501 20:00:33.304230 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:33.304242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:33.304312 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:33.304757 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:33.304763 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:33.304784 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:33.305030 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:33.305037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.306093 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:33.306096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:33.306860 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:33.306870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.306970 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:33.307023 (+ 53us) client_negotiation.cc:769] Sending connection context
0501 20:00:33.307109 (+ 86us) client_negotiation.cc:240] Negotiation successful
0501 20:00:33.307188 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"thread_start_us":85,"threads_started":1}
I20240501 20:00:33.307794 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:33.303198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:35715 (local address 127.0.0.1:46578)
0501 20:00:33.303906 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:33.303945 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:33.304072 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:33.304819 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:33.304822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:33.304841 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:33.305092 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:33.305100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.306476 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:33.306479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:33.307302 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:33.307310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.307401 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:33.307416 (+ 15us) client_negotiation.cc:769] Sending connection context
0501 20:00:33.307630 (+ 214us) client_negotiation.cc:240] Negotiation successful
0501 20:00:33.307686 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":101,"threads_started":1}
I20240501 20:00:33.308130 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:33.303490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:34983 (local address 127.0.0.1:51050)
0501 20:00:33.304539 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:33.304551 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:33.304619 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:33.305253 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:33.305256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:33.305275 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:33.305506 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:33.305513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.306903 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:33.306906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:33.307776 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:33.307786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.307887 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:33.307903 (+ 16us) client_negotiation.cc:769] Sending connection context
0501 20:00:33.307946 (+ 43us) client_negotiation.cc:240] Negotiation successful
0501 20:00:33.308010 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"thread_start_us":75,"threads_started":1}
I20240501 20:00:33.308151 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:33.303947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42286 (local address 127.8.168.66:37011)
0501 20:00:33.304311 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:33.304318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:33.304343 (+ 25us) server_negotiation.cc:407] Connection header received
0501 20:00:33.304433 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:33.304438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:33.304578 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:33.304712 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:33.305138 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.305999 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:33.307210 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.307751 (+ 541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:33.307874 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:33.307942 (+ 68us) server_negotiation.cc:299] Negotiation successful
0501 20:00:33.308029 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":107,"threads_started":1}
I20240501 20:00:33.308876 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:33.303330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46578 (local address 127.8.168.65:35715)
0501 20:00:33.304354 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:33.304360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:33.304381 (+ 21us) server_negotiation.cc:407] Connection header received
0501 20:00:33.304478 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:33.304483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:33.304654 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:33.305232 (+ 578us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:33.305365 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.306365 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:33.308060 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.308514 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:33.308595 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:33.308651 (+ 56us) server_negotiation.cc:299] Negotiation successful
0501 20:00:33.308714 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":930,"thread_start_us":103,"threads_started":1}
I20240501 20:00:33.309401 24186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:33.303810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51050 (local address 127.8.168.67:34983)
0501 20:00:33.304873 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:33.304879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:33.304898 (+ 19us) server_negotiation.cc:407] Connection header received
0501 20:00:33.304986 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:33.304991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:33.305130 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:33.305256 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:33.305622 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.306791 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:33.308360 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.308894 (+ 534us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:33.309006 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:33.309076 (+ 70us) server_negotiation.cc:299] Negotiation successful
0501 20:00:33.309155 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":970,"thread_start_us":132,"threads_started":1}
I20240501 20:00:33.310988 23971 tablet_service.cc:1467] Processing CreateTablet for tablet 8d02484c25e54c09b8e4ec4a12bfce70 (DEFAULT_TABLE table=test-table [id=82d2e4e5dbb74ee7971ec03b9c156a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:33.311641 23837 tablet_service.cc:1467] Processing CreateTablet for tablet 8d02484c25e54c09b8e4ec4a12bfce70 (DEFAULT_TABLE table=test-table [id=82d2e4e5dbb74ee7971ec03b9c156a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:33.312199 23971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d02484c25e54c09b8e4ec4a12bfce70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.311796 24105 tablet_service.cc:1467] Processing CreateTablet for tablet 8d02484c25e54c09b8e4ec4a12bfce70 (DEFAULT_TABLE table=test-table [id=82d2e4e5dbb74ee7971ec03b9c156a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:33.312472 23837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d02484c25e54c09b8e4ec4a12bfce70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.312801 24105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d02484c25e54c09b8e4ec4a12bfce70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.317878 24188 tablet_bootstrap.cc:492] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259: Bootstrap starting.
I20240501 20:00:33.319275 24189 tablet_bootstrap.cc:492] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572: Bootstrap starting.
I20240501 20:00:33.319381 24190 tablet_bootstrap.cc:492] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697: Bootstrap starting.
I20240501 20:00:33.320544 24188 tablet_bootstrap.cc:654] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:33.321214 24190 tablet_bootstrap.cc:654] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:33.321214 24189 tablet_bootstrap.cc:654] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:33.321643 24188 log.cc:824] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:33.322016 24189 log.cc:824] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:33.322309 24190 log.cc:824] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:33.323431 24188 tablet_bootstrap.cc:492] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259: No bootstrap required, opened a new log
I20240501 20:00:33.323604 24188 ts_tablet_manager.cc:1367] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:00:33.323657 24189 tablet_bootstrap.cc:492] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572: No bootstrap required, opened a new log
I20240501 20:00:33.323814 24189 ts_tablet_manager.cc:1367] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:00:33.324271 24190 tablet_bootstrap.cc:492] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697: No bootstrap required, opened a new log
I20240501 20:00:33.324651 24190 ts_tablet_manager.cc:1367] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:00:33.326532 24189 raft_consensus.cc:348] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } } peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } }
I20240501 20:00:33.326718 24189 raft_consensus.cc:374] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:33.326814 24189 raft_consensus.cc:729] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa7730f481b4205b2813adf412b5572, State: Initialized, Role: FOLLOWER
I20240501 20:00:33.326767 24188 raft_consensus.cc:348] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } } peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } }
I20240501 20:00:33.327075 24188 raft_consensus.cc:374] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:33.327160 24188 raft_consensus.cc:729] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ba60fb865a42c8bdf5b26b2e647259, State: Initialized, Role: FOLLOWER
I20240501 20:00:33.327329 24189 consensus_queue.cc:260] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } } peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } }
I20240501 20:00:33.327622 24188 consensus_queue.cc:260] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } } peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } }
I20240501 20:00:33.327876 24190 raft_consensus.cc:348] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } } peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } }
I20240501 20:00:33.328086 24190 raft_consensus.cc:374] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:33.328109 24171 heartbeater.cc:503] Master 127.8.168.126:39523 was elected leader, sending a full tablet report...
I20240501 20:00:33.328143 24190 raft_consensus.cc:729] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a823d5a9c1134ecfae1bed89bce5c697, State: Initialized, Role: FOLLOWER
I20240501 20:00:33.328297 24189 ts_tablet_manager.cc:1398] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:00:33.328513 24190 consensus_queue.cc:260] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } } peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } }
I20240501 20:00:33.329123 23903 heartbeater.cc:503] Master 127.8.168.126:39523 was elected leader, sending a full tablet report...
I20240501 20:00:33.329326 24188 ts_tablet_manager.cc:1398] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 20:00:33.329325 24190 ts_tablet_manager.cc:1398] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20240501 20:00:33.329667 24037 heartbeater.cc:503] Master 127.8.168.126:39523 was elected leader, sending a full tablet report...
I20240501 20:00:33.347599 24194 raft_consensus.cc:482] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:33.347785 24194 raft_consensus.cc:504] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } } peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } }
I20240501 20:00:33.348819 24194 leader_election.cc:288] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8ba60fb865a42c8bdf5b26b2e647259 (127.8.168.65:35715), a823d5a9c1134ecfae1bed89bce5c697 (127.8.168.66:37011)
I20240501 20:00:33.351677 24173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:33.348994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:37011 (local address 127.8.168.67:48137)
0501 20:00:33.349124 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:33.349139 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:33.349227 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:33.349513 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:33.349516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:33.349537 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:33.349799 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:33.349806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.350733 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:33.350737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:33.351309 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:33.351317 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.351422 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:33.351444 (+ 22us) client_negotiation.cc:769] Sending connection context
0501 20:00:33.351493 (+ 49us) client_negotiation.cc:240] Negotiation successful
0501 20:00:33.351547 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20240501 20:00:33.351811 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:33.349011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:35715 (local address 127.8.168.67:37969)
0501 20:00:33.349587 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:33.349599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:33.349687 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:33.349908 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:33.349911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:33.349925 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:33.350123 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:33.350129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.350801 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:33.350805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:33.351378 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:33.351386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.351496 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:33.351514 (+ 18us) client_negotiation.cc:769] Sending connection context
0501 20:00:33.351562 (+ 48us) client_negotiation.cc:240] Negotiation successful
0501 20:00:33.351612 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":233,"thread_start_us":156,"threads_started":1}
I20240501 20:00:33.352133 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:33.349146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:48137 (local address 127.8.168.66:37011)
0501 20:00:33.349278 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:33.349282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:33.349297 (+ 15us) server_negotiation.cc:407] Connection header received
0501 20:00:33.349349 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:33.349352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:33.349406 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:33.349488 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:33.350010 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.350627 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:33.351421 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.351887 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:33.351926 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:33.351976 (+ 50us) server_negotiation.cc:299] Negotiation successful
0501 20:00:33.352022 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240501 20:00:33.352200 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:33.349197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:37969 (local address 127.8.168.65:35715)
0501 20:00:33.349331 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:33.349334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:33.349676 (+ 342us) server_negotiation.cc:407] Connection header received
0501 20:00:33.349765 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:33.349767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:33.349820 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:33.349890 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:33.350228 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.350701 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:33.351486 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.351927 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:33.351965 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:33.352034 (+ 69us) server_negotiation.cc:299] Negotiation successful
0501 20:00:33.352097 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240501 20:00:33.352684 23857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d02484c25e54c09b8e4ec4a12bfce70" candidate_uuid: "5aa7730f481b4205b2813adf412b5572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" is_pre_election: true
I20240501 20:00:33.352679 23991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d02484c25e54c09b8e4ec4a12bfce70" candidate_uuid: "5aa7730f481b4205b2813adf412b5572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a823d5a9c1134ecfae1bed89bce5c697" is_pre_election: true
I20240501 20:00:33.352993 23857 raft_consensus.cc:2454] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa7730f481b4205b2813adf412b5572 in term 0.
I20240501 20:00:33.352993 23991 raft_consensus.cc:2454] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa7730f481b4205b2813adf412b5572 in term 0.
I20240501 20:00:33.353432 24058 leader_election.cc:302] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa7730f481b4205b2813adf412b5572, a823d5a9c1134ecfae1bed89bce5c697; no voters:
I20240501 20:00:33.353680 24194 raft_consensus.cc:2781] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:33.353746 24194 raft_consensus.cc:482] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:33.353775 24194 raft_consensus.cc:3037] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:33.354682 24194 raft_consensus.cc:504] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } } peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } }
I20240501 20:00:33.355010 24194 leader_election.cc:288] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [CANDIDATE]: Term 1 election: Requested vote from peers b8ba60fb865a42c8bdf5b26b2e647259 (127.8.168.65:35715), a823d5a9c1134ecfae1bed89bce5c697 (127.8.168.66:37011)
I20240501 20:00:33.355340 23857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d02484c25e54c09b8e4ec4a12bfce70" candidate_uuid: "5aa7730f481b4205b2813adf412b5572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8ba60fb865a42c8bdf5b26b2e647259"
I20240501 20:00:33.355356 23991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d02484c25e54c09b8e4ec4a12bfce70" candidate_uuid: "5aa7730f481b4205b2813adf412b5572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a823d5a9c1134ecfae1bed89bce5c697"
I20240501 20:00:33.355470 23991 raft_consensus.cc:3037] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:33.355455 23857 raft_consensus.cc:3037] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:33.356500 23991 raft_consensus.cc:2454] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa7730f481b4205b2813adf412b5572 in term 1.
I20240501 20:00:33.356513 23857 raft_consensus.cc:2454] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa7730f481b4205b2813adf412b5572 in term 1.
I20240501 20:00:33.356801 24058 leader_election.cc:302] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa7730f481b4205b2813adf412b5572, a823d5a9c1134ecfae1bed89bce5c697; no voters:
I20240501 20:00:33.356950 24194 raft_consensus.cc:2781] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:33.357184 24194 raft_consensus.cc:686] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [term 1 LEADER]: Becoming Leader. State: Replica: 5aa7730f481b4205b2813adf412b5572, State: Running, Role: LEADER
I20240501 20:00:33.357477 24194 consensus_queue.cc:237] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } } peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } }
I20240501 20:00:33.360448 23712 catalog_manager.cc:5484] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5aa7730f481b4205b2813adf412b5572 (127.8.168.67). New cstate: current_term: 1 leader_uuid: "5aa7730f481b4205b2813adf412b5572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } health_report { overall_health: HEALTHY } } }
W20240501 20:00:33.371349 23904 tablet.cc:2369] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:00:33.375487 24186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:33.372556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51054 (local address 127.8.168.67:34983)
0501 20:00:33.372681 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:33.372685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:33.372698 (+ 13us) server_negotiation.cc:407] Connection header received
0501 20:00:33.372798 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:33.372801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:33.372855 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:33.372952 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:00:33.373266 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.373762 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:33.374386 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:33.374566 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:33.374712 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:00:33.375123 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:00:33.375232 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:33.375308 (+ 76us) server_negotiation.cc:299] Negotiation successful
0501 20:00:33.375358 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240501 20:00:33.380621 23991 raft_consensus.cc:1261] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697 [term 1 FOLLOWER]: Refusing update from remote peer 5aa7730f481b4205b2813adf412b5572: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:00:33.380663 23857 raft_consensus.cc:1261] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Refusing update from remote peer 5aa7730f481b4205b2813adf412b5572: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:00:33.381320 24198 consensus_queue.cc:1035] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:33.381484 24194 consensus_queue.cc:1035] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:33.387198 24201 mvcc.cc:205] Tried to move back new op lower bound from 7022975522320252928 to 7022975522234851328. Current Snapshot: MvccSnapshot[applied={T|T < 7022975522320252928}]
I20240501 20:00:33.395449 23712 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52806:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:00:33.395606 23712 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52806:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:00:33.397876 23712 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: Sending DeleteTablet for 3 replicas of tablet 8d02484c25e54c09b8e4ec4a12bfce70
I20240501 20:00:33.398641 23837 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d02484c25e54c09b8e4ec4a12bfce70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:00:33 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:46578
I20240501 20:00:33.398764 23971 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d02484c25e54c09b8e4ec4a12bfce70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:00:33 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:42286
I20240501 20:00:33.398942 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 23774
I20240501 20:00:33.399112 24105 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d02484c25e54c09b8e4ec4a12bfce70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:00:33 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:51050
I20240501 20:00:33.401213 24211 tablet_replica.cc:330] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697: stopping tablet replica
I20240501 20:00:33.401476 24211 raft_consensus.cc:2229] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:33.401782 24211 raft_consensus.cc:2258] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:33.403110 24211 ts_tablet_manager.cc:1875] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:00:33.405670 24211 ts_tablet_manager.cc:1888] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:00:33.405766 24211 log.cc:1196] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/8d02484c25e54c09b8e4ec4a12bfce70
I20240501 20:00:33.406042 24212 tablet_replica.cc:330] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572: stopping tablet replica
I20240501 20:00:33.406150 24211 ts_tablet_manager.cc:1909] T 8d02484c25e54c09b8e4ec4a12bfce70 P a823d5a9c1134ecfae1bed89bce5c697: Deleting consensus metadata
I20240501 20:00:33.406355 24212 raft_consensus.cc:2229] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [term 1 LEADER]: Raft consensus shutting down.
W20240501 20:00:33.406677 23699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.168.65:35715 (error 108)
I20240501 20:00:33.406821 24212 raft_consensus.cc:2258] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:33.407474 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 23908
I20240501 20:00:33.407660 23696 catalog_manager.cc:4830] TS a823d5a9c1134ecfae1bed89bce5c697 (127.8.168.66:37011): tablet 8d02484c25e54c09b8e4ec4a12bfce70 (table test-table [id=82d2e4e5dbb74ee7971ec03b9c156a22]) successfully deleted
I20240501 20:00:33.408977 24212 ts_tablet_manager.cc:1875] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:00:33.409122 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:33.408802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:35715 (local address 127.0.0.1:46592)
0501 20:00:33.408972 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:33.409034 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:35715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20240501 20:00:33.409343 23699 catalog_manager.cc:4573] TS b8ba60fb865a42c8bdf5b26b2e647259 (127.8.168.65:35715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d02484c25e54c09b8e4ec4a12bfce70: Network error: Client connection negotiation failed: client connection to 127.8.168.65:35715: connect: Connection refused (error 111)
I20240501 20:00:33.411437 24212 ts_tablet_manager.cc:1888] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:00:33.411535 24212 log.cc:1196] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/8d02484c25e54c09b8e4ec4a12bfce70
I20240501 20:00:33.411882 24212 ts_tablet_manager.cc:1909] T 8d02484c25e54c09b8e4ec4a12bfce70 P 5aa7730f481b4205b2813adf412b5572: Deleting consensus metadata
I20240501 20:00:33.412918 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 24042
I20240501 20:00:33.412994 23698 catalog_manager.cc:4830] TS 5aa7730f481b4205b2813adf412b5572 (127.8.168.67:34983): tablet 8d02484c25e54c09b8e4ec4a12bfce70 (table test-table [id=82d2e4e5dbb74ee7971ec03b9c156a22]) successfully deleted
I20240501 20:00:33.417342 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 23680
I20240501 20:00:53.422622 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:39523
--webserver_interface=127.8.168.126
--webserver_port=41413
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:39523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:00:53.525925 24214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:53.526166 24214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:53.526217 24214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:53.529304 24214 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20240501 20:00:53.529356 24214 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20240501 20:00:53.529373 24214 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20240501 20:00:53.529386 24214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:00:53.529404 24214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:53.529424 24214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:53.529444 24214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:00:53.529462 24214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:00:53.533483 24214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:39523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:39523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=41413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:53.534327 24214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:53.535112 24214 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:53.546222 24223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:53.546591 24221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:53.546893 24220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:53.547441 24214 server_base.cc:1034] running on GCE node
I20240501 20:00:53.547981 24214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:53.548945 24214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:53.550109 24214 hybrid_clock.cc:648] HybridClock initialized: now 1714593653550092 us; error 27 us; skew 500 ppm
May 01 20:00:53 dist-test-slave-tzct krb5kdc[23667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593653, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:53.552924 24214 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:00:53.553864 24214 webserver.cc:460] Webserver started at http://127.8.168.126:41413/ using document root <none> and password file <none>
I20240501 20:00:53.554430 24214 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:53.554504 24214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:53.558072 24214 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:53.560158 24230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:53.561221 24214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:53.561336 24214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "d43c9a64005e41efab8421ad002ddf76"
format_stamp: "Formatted at 2024-05-01 20:00:32 on dist-test-slave-tzct"
server_key: "6151a5281cdbf86d42ce39f35731b5ec"
server_key_iv: "e8baaa7c478998be38e53b28cf4612d3"
server_key_version: "encryptionkey@0"
I20240501 20:00:53.561836 24214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:53.582047 24214 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:53.584797 24214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:53.584957 24214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:53.591428 24214 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:39523
I20240501 20:00:53.591645 24282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:39523 every 8 connection(s)
I20240501 20:00:53.592502 24214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:00:53.596329 24283 sys_catalog.cc:263] Verifying existing consensus state
I20240501 20:00:53.597843 24283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: Bootstrap starting.
I20240501 20:00:53.598199 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 24214
I20240501 20:00:53.598651 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:35715
--local_ip_for_outbound_sockets=127.8.168.65
--tserver_master_addrs=127.8.168.126:39523
--webserver_port=35677
--webserver_interface=127.8.168.65
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 20:00:53.614143 24283 log.cc:824] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:53.622187 24283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:00:53.622498 24283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: Bootstrap complete.
I20240501 20:00:53.625761 24283 raft_consensus.cc:348] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } }
I20240501 20:00:53.626081 24283 raft_consensus.cc:729] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43c9a64005e41efab8421ad002ddf76, State: Initialized, Role: FOLLOWER
I20240501 20:00:53.626534 24283 consensus_queue.cc:260] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } }
I20240501 20:00:53.626658 24283 raft_consensus.cc:388] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:00:53.626719 24283 raft_consensus.cc:482] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:00:53.626821 24283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:53.628306 24283 raft_consensus.cc:504] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } }
I20240501 20:00:53.628600 24283 leader_election.cc:302] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43c9a64005e41efab8421ad002ddf76; no voters:
I20240501 20:00:53.628885 24283 leader_election.cc:288] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240501 20:00:53.629004 24287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:53.629201 24287 raft_consensus.cc:686] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 2 LEADER]: Becoming Leader. State: Replica: d43c9a64005e41efab8421ad002ddf76, State: Running, Role: LEADER
I20240501 20:00:53.629529 24287 consensus_queue.cc:237] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } }
I20240501 20:00:53.629976 24283 sys_catalog.cc:564] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:00:53.630749 24288 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d43c9a64005e41efab8421ad002ddf76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } } }
I20240501 20:00:53.630867 24288 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:53.631170 24289 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d43c9a64005e41efab8421ad002ddf76. Latest consensus state: current_term: 2 leader_uuid: "d43c9a64005e41efab8421ad002ddf76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } } }
I20240501 20:00:53.631292 24289 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:53.631546 24296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:00:53.635859 24296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:00:53.636699 24296 catalog_manager.cc:1259] Loaded cluster ID: c53a43c37cb146a5a2f0ad2dd9359472
I20240501 20:00:53.636776 24296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:00:53.639153 24296 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:00:53.639838 24296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: Loaded TSK: 0
I20240501 20:00:53.653530 24296 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: Generated new TSK 1
I20240501 20:00:53.653784 24296 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240501 20:00:53.711081 24285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:53.711331 24285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:53.711392 24285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:53.714584 24285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:53.714656 24285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:53.714759 24285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 20:00:53.718931 24285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:35715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=35677
--tserver_master_addrs=127.8.168.126:39523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:53.719815 24285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:53.720654 24285 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:53.728849 24310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:53.729202 24313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:53.729415 24311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:53.729967 24285 server_base.cc:1034] running on GCE node
I20240501 20:00:53.730341 24285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:53.730939 24285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:53.732096 24285 hybrid_clock.cc:648] HybridClock initialized: now 1714593653732056 us; error 90 us; skew 500 ppm
May 01 20:00:53 dist-test-slave-tzct krb5kdc[23667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593653, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:53.734941 24285 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 20:00:53.735855 24285 webserver.cc:460] Webserver started at http://127.8.168.65:35677/ using document root <none> and password file <none>
I20240501 20:00:53.736363 24285 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:53.736439 24285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:53.739969 24285 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:53.742136 24320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:53.743294 24285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:00:53.743410 24285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "b8ba60fb865a42c8bdf5b26b2e647259"
format_stamp: "Formatted at 2024-05-01 20:00:32 on dist-test-slave-tzct"
server_key: "5886a6aaaffc4a84ad0c7c9bd276ac9a"
server_key_iv: "e9c39f5e79127df8c784460527f53ff1"
server_key_version: "encryptionkey@0"
I20240501 20:00:53.743857 24285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:53.763607 24285 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:53.766568 24285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:53.766722 24285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:53.767303 24285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:53.768396 24327 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 20:00:53.772903 24285 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 20:00:53.772966 24285 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20240501 20:00:53.773030 24285 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 20:00:53.774533 24285 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 20:00:53.774582 24285 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240501 20:00:53.774852 24327 tablet_bootstrap.cc:492] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259: Bootstrap starting.
I20240501 20:00:53.783977 24285 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:35715
I20240501 20:00:53.784216 24434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:35715 every 8 connection(s)
I20240501 20:00:53.785089 24285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 20:00:53.793531 24327 log.cc:824] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259: Log is configured to *not* fsync() on all Append() calls
May 01 20:00:53 dist-test-slave-tzct krb5kdc[23667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593653, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:53.794687 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 24285
I20240501 20:00:53.795138 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:37011
--local_ip_for_outbound_sockets=127.8.168.66
--tserver_master_addrs=127.8.168.126:39523
--webserver_port=37473
--webserver_interface=127.8.168.66
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 20:00:53.799408 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:53.786827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:50891)
0501 20:00:53.787464 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:53.787504 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:53.788217 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:53.789227 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:53.789236 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:53.789645 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:53.790155 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:53.790169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:53.791182 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:53.791185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:53.791573 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:53.791580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:53.791745 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:53.792459 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:53.792486 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:53.794132 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:53.796631 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:53.796637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:53.796647 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:53.796970 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:53.797304 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:53.797307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:53.797310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:53.797411 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:53.797978 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:53.797984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:53.798248 (+ 264us) client_negotiation.cc:769] Sending connection context
0501 20:00:53.798449 (+ 201us) client_negotiation.cc:240] Negotiation successful
0501 20:00:53.798719 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":113,"threads_started":1}
I20240501 20:00:53.799515 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:53.786846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:50891 (local address 127.8.168.126:39523)
0501 20:00:53.787201 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:53.787213 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:53.787606 (+ 393us) server_negotiation.cc:407] Connection header received
0501 20:00:53.788590 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:53.788600 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:53.788872 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:53.789164 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:53.790305 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:53.791061 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:53.791683 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:53.791927 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:53.794328 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:53.794349 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:53.794361 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:53.794397 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:53.796499 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:53.797083 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:53.797089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:53.797095 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:53.797201 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:53.797495 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:53.797498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:53.797500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:53.797858 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:53.797992 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:53.798592 (+ 600us) server_negotiation.cc:299] Negotiation successful
0501 20:00:53.798865 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":109,"threads_started":1}
I20240501 20:00:53.800474 24327 tablet_bootstrap.cc:492] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:00:53.800813 24327 tablet_bootstrap.cc:492] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259: Bootstrap complete.
I20240501 20:00:53.801947 24435 heartbeater.cc:347] Connected to a master server at 127.8.168.126:39523
I20240501 20:00:53.801986 24327 ts_tablet_manager.cc:1367] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.013s
I20240501 20:00:53.802187 24435 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:53.802803 24435 heartbeater.cc:511] Master 127.8.168.126:39523 requested a full tablet report, sending...
I20240501 20:00:53.803891 24327 raft_consensus.cc:348] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } } peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } }
I20240501 20:00:53.804292 24327 raft_consensus.cc:729] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ba60fb865a42c8bdf5b26b2e647259, State: Initialized, Role: FOLLOWER
I20240501 20:00:53.804764 24327 consensus_queue.cc:260] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } } peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } }
I20240501 20:00:53.805434 24247 ts_manager.cc:194] Registered new tserver with Master: b8ba60fb865a42c8bdf5b26b2e647259 (127.8.168.65:35715)
I20240501 20:00:53.805603 24327 ts_tablet_manager.cc:1398] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:00:53.808588 24247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:50891
I20240501 20:00:53.810993 24435 heartbeater.cc:503] Master 127.8.168.126:39523 was elected leader, sending a full tablet report...
May 01 20:00:53 dist-test-slave-tzct krb5kdc[23667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593653, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for kudu/127.8.168.65@KRBTEST.COM
I20240501 20:00:53.817152 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:53.807914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:35715 (local address 127.0.0.1:36026)
0501 20:00:53.808288 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:53.808328 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:53.808451 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:53.809033 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:53.809039 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:53.809507 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:53.809774 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:53.809783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:53.810938 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:53.810942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:53.811460 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:53.811471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:53.811572 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:53.812205 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:53.812220 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:53.813795 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:53.815570 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:53.815578 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:53.815582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:53.815837 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:53.816055 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:53.816057 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:53.816059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:53.816112 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:53.816554 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:53.816559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:53.816668 (+ 109us) client_negotiation.cc:769] Sending connection context
0501 20:00:53.816839 (+ 171us) client_negotiation.cc:240] Negotiation successful
0501 20:00:53.816989 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":102,"threads_started":1}
I20240501 20:00:53.817168 24443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:53.807992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36026 (local address 127.8.168.65:35715)
0501 20:00:53.808566 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:53.808572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:53.808593 (+ 21us) server_negotiation.cc:407] Connection header received
0501 20:00:53.808667 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:53.808672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:53.808871 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:53.809003 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:53.809898 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:53.810822 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:53.811835 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:53.812053 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:53.813915 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:53.813935 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:53.813943 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:53.813964 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:53.815463 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:53.815933 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:53.815937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:53.815938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:53.815983 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:53.816266 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:53.816268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:53.816270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:53.816448 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:53.816607 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:53.816923 (+ 316us) server_negotiation.cc:299] Negotiation successful
0501 20:00:53.817047 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":95,"threads_started":1}
I20240501 20:00:53.818044 24369 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d02484c25e54c09b8e4ec4a12bfce70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:00:33 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:36026
I20240501 20:00:53.818774 24444 tablet_replica.cc:330] stopping tablet replica
I20240501 20:00:53.819056 24444 raft_consensus.cc:2229] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:53.819304 24444 raft_consensus.cc:2258] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:53.820276 24444 ts_tablet_manager.cc:1875] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:00:53.823931 24444 ts_tablet_manager.cc:1888] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:00:53.824018 24444 log.cc:1196] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal/wals/8d02484c25e54c09b8e4ec4a12bfce70
I20240501 20:00:53.824301 24444 ts_tablet_manager.cc:1909] T 8d02484c25e54c09b8e4ec4a12bfce70 P b8ba60fb865a42c8bdf5b26b2e647259: Deleting consensus metadata
I20240501 20:00:53.825992 24234 catalog_manager.cc:4830] TS b8ba60fb865a42c8bdf5b26b2e647259 (127.8.168.65:35715): tablet 8d02484c25e54c09b8e4ec4a12bfce70 (table test-table [id=82d2e4e5dbb74ee7971ec03b9c156a22]) successfully deleted
W20240501 20:00:53.904165 24440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:53.904374 24440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:53.904426 24440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:53.907393 24440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:53.907451 24440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:53.907524 24440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 20:00:53.911448 24440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:37011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=37473
--tserver_master_addrs=127.8.168.126:39523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:53.912230 24440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:53.913033 24440 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:53.920835 24452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:53.921016 24450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:53.921464 24449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:53.921907 24440 server_base.cc:1034] running on GCE node
I20240501 20:00:53.922241 24440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:53.922734 24440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:53.923883 24440 hybrid_clock.cc:648] HybridClock initialized: now 1714593653923871 us; error 31 us; skew 500 ppm
May 01 20:00:53 dist-test-slave-tzct krb5kdc[23667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593653, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:53.926509 24440 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 20:00:53.927369 24440 webserver.cc:460] Webserver started at http://127.8.168.66:37473/ using document root <none> and password file <none>
I20240501 20:00:53.927879 24440 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:53.927949 24440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:53.931478 24440 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:53.933333 24459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:53.934295 24440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:53.934394 24440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "a823d5a9c1134ecfae1bed89bce5c697"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-tzct"
server_key: "5342e2f07c3fc12fc5f8584fe31b9d40"
server_key_iv: "16c93e65cb43f98f3178d77bdcb7cdc6"
server_key_version: "encryptionkey@0"
I20240501 20:00:53.934785 24440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:53.950838 24440 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:53.953312 24440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:53.953460 24440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:53.953938 24440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:53.954667 24440 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:53.954712 24440 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:53.954770 24440 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:53.954797 24440 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:53.962795 24440 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:37011
I20240501 20:00:53.962831 24572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:37011 every 8 connection(s)
I20240501 20:00:53.963752 24440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 20:00:53.971336 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 24440
May 01 20:00:53 dist-test-slave-tzct krb5kdc[23667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593653, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:53.971752 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:34983
--local_ip_for_outbound_sockets=127.8.168.67
--tserver_master_addrs=127.8.168.126:39523
--webserver_port=41535
--webserver_interface=127.8.168.67
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 20:00:53.975796 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:53.965343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:36753 (local address 127.8.168.126:39523)
0501 20:00:53.965505 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:53.965510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:53.966178 (+ 668us) server_negotiation.cc:407] Connection header received
0501 20:00:53.967377 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:53.967380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:53.967429 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:53.967501 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:53.968813 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:53.969299 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:53.969909 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:53.970030 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:53.972193 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:53.972217 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:53.972220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:53.972259 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:53.973828 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:53.974383 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:53.974386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:53.974387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:53.974426 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:53.974696 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:53.974698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:53.974699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:53.974862 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:53.974949 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:53.975561 (+ 612us) server_negotiation.cc:299] Negotiation successful
0501 20:00:53.975672 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20240501 20:00:53.976701 24575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:53.965634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:36753)
0501 20:00:53.966051 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:53.966088 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:53.967190 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:53.967702 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:53.967711 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:53.968074 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:53.968671 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:53.968685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:53.969397 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:53.969400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:53.969824 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:53.969832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:53.970001 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:53.970555 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:53.970589 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:53.972026 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:53.973947 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:53.973953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:53.973963 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:53.974291 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:53.974491 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:53.974494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:53.974496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:53.974619 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:53.974971 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:53.974977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:53.975296 (+ 319us) client_negotiation.cc:769] Sending connection context
0501 20:00:53.975534 (+ 238us) client_negotiation.cc:240] Negotiation successful
0501 20:00:53.975848 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":91,"threads_started":1}
I20240501 20:00:53.977844 24573 heartbeater.cc:347] Connected to a master server at 127.8.168.126:39523
I20240501 20:00:53.978051 24573 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:53.978539 24573 heartbeater.cc:511] Master 127.8.168.126:39523 requested a full tablet report, sending...
I20240501 20:00:53.979533 24247 ts_manager.cc:194] Registered new tserver with Master: a823d5a9c1134ecfae1bed89bce5c697 (127.8.168.66:37011)
I20240501 20:00:53.980289 24247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:36753
W20240501 20:00:54.075881 24576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:00:54.076114 24576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:00:54.076169 24576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:00:54.079159 24576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:00:54.079216 24576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:00:54.079288 24576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 20:00:54.083315 24576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:34983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=41535
--tserver_master_addrs=127.8.168.126:39523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:00:54.084156 24576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:54.084944 24576 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:54.092307 24584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:54.092813 24581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:54.093164 24582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:54.093705 24576 server_base.cc:1034] running on GCE node
I20240501 20:00:54.094030 24576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:00:54.094635 24576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:00:54.095793 24576 hybrid_clock.cc:648] HybridClock initialized: now 1714593654095776 us; error 37 us; skew 500 ppm
May 01 20:00:54 dist-test-slave-tzct krb5kdc[23667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593654, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:00:54.098280 24576 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 20:00:54.099126 24576 webserver.cc:460] Webserver started at http://127.8.168.67:41535/ using document root <none> and password file <none>
I20240501 20:00:54.099620 24576 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:54.099686 24576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:54.102958 24576 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:00:54.104679 24591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:54.105682 24576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:00:54.105779 24576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "5aa7730f481b4205b2813adf412b5572"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-tzct"
server_key: "465d77d6537b7b7d8f82505574f85fb8"
server_key_iv: "5462dfd0b99d45605b53340821222c1a"
server_key_version: "encryptionkey@0"
I20240501 20:00:54.106174 24576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:54.118378 24576 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:54.121142 24576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:54.121289 24576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:54.121776 24576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:54.122524 24576 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:54.122571 24576 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:54.122614 24576 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:54.122653 24576 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:54.130643 24576 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:34983
I20240501 20:00:54.130715 24704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:34983 every 8 connection(s)
I20240501 20:00:54.131675 24576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 20:00:54.136920 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 24576
May 01 20:00:54 dist-test-slave-tzct krb5kdc[23667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593654, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:00:54.143276 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:54.133235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:42477 (local address 127.8.168.126:39523)
0501 20:00:54.133354 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:54.133357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:54.133966 (+ 609us) server_negotiation.cc:407] Connection header received
0501 20:00:54.135123 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:54.135126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:54.135187 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:54.135258 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:54.136460 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.137110 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:54.137880 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.138069 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:54.140127 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:54.140144 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:54.140146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:54.140168 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:54.141512 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:54.141953 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:54.141956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:54.141957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:54.141999 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:54.142233 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:54.142237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:54.142239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:54.142376 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:54.142458 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:54.142921 (+ 463us) server_negotiation.cc:299] Negotiation successful
0501 20:00:54.143116 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240501 20:00:54.143787 24707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:54.133477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:42477)
0501 20:00:54.133847 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:54.133879 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:54.134928 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:54.135419 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:54.135426 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:54.135772 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:00:54.136324 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:54.136338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.137228 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:54.137231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:54.137779 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:54.137789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.137978 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:54.138627 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:00:54.138650 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:00:54.139986 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:00:54.141616 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:54.141622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:54.141632 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:54.141864 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:54.142074 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:00:54.142077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:00:54.142079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:00:54.142155 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:00:54.142472 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:00:54.142478 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:00:54.142690 (+ 212us) client_negotiation.cc:769] Sending connection context
0501 20:00:54.142868 (+ 178us) client_negotiation.cc:240] Negotiation successful
0501 20:00:54.143117 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":95,"threads_started":1}
I20240501 20:00:54.144811 24705 heartbeater.cc:347] Connected to a master server at 127.8.168.126:39523
I20240501 20:00:54.145017 24705 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:54.145030 24709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:54.137563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56270 (local address 127.8.168.126:39523)
0501 20:00:54.137768 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:54.137772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:54.137901 (+ 129us) server_negotiation.cc:407] Connection header received
0501 20:00:54.138014 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:54.138017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:54.138089 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:54.138171 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:54.139116 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.139629 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:54.140264 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.140447 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:54.141679 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:54.141703 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:54.141706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:54.141732 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:54.143118 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:54.143690 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:54.143694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:54.143696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:54.143868 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:54.144218 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:54.144221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:54.144223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:54.144384 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:54.144564 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:54.144712 (+ 148us) server_negotiation.cc:299] Negotiation successful
0501 20:00:54.144876 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
I20240501 20:00:54.145496 24705 heartbeater.cc:511] Master 127.8.168.126:39523 requested a full tablet report, sending...
I20240501 20:00:54.146394 24247 ts_manager.cc:194] Registered new tserver with Master: 5aa7730f481b4205b2813adf412b5572 (127.8.168.67:34983)
I20240501 20:00:54.146903 24247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:42477
I20240501 20:00:54.148412 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 20:00:54.151580 24709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:54.148947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56282 (local address 127.8.168.126:39523)
0501 20:00:54.149051 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:54.149055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:54.149227 (+ 172us) server_negotiation.cc:407] Connection header received
0501 20:00:54.149377 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:54.149379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:54.149453 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:54.149541 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:00:54.149872 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.150322 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:54.150911 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.151070 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:54.151143 (+ 73us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:00:54.151264 (+ 121us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0501 20:00:54.151473 (+ 209us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56282: token expired
Metrics: {"server-negotiator.queue_time_us":35}
W20240501 20:00:54.151633 24709 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56282: token expired
W20240501 20:00:54.151875 24176 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.168.126:39523)
I20240501 20:00:54.157709 24709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:54.152180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56290 (local address 127.8.168.126:39523)
0501 20:00:54.152284 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:54.152286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:54.152364 (+ 78us) server_negotiation.cc:407] Connection header received
0501 20:00:54.152457 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:54.152459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:54.152491 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:54.152550 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:00:54.153185 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.153618 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:54.154120 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.154227 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:54.155332 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:00:54.155348 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:00:54.155351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:00:54.155376 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:00:54.156489 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:54.156851 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:54.156854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:54.156856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:54.156906 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:00:54.157083 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:00:54.157086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:00:54.157087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:00:54.157216 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:00:54.157318 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:54.157444 (+ 126us) server_negotiation.cc:299] Negotiation successful
0501 20:00:54.157548 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240501 20:00:54.159997 24247 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:00:54.162897 24247 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:00:54.169975 24369 tablet_service.cc:1467] Processing CreateTablet for tablet 9cdf00559c6c495bb53dabbb4ee633f4 (DEFAULT_TABLE table=test-table [id=794d65fe1fa24c3abe9a875c248c5ded]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:54.170500 24369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cdf00559c6c495bb53dabbb4ee633f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:54.172899 24327 tablet_bootstrap.cc:492] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259: Bootstrap starting.
I20240501 20:00:54.173367 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:54.169470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:34983 (local address 127.0.0.1:52164)
0501 20:00:54.169650 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:54.169664 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:54.169741 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:54.170507 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:54.170510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:54.170528 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:54.170776 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:54.170785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.172058 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:54.172062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:54.172960 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:54.172970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.173066 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:54.173118 (+ 52us) client_negotiation.cc:769] Sending connection context
0501 20:00:54.173167 (+ 49us) client_negotiation.cc:240] Negotiation successful
0501 20:00:54.173236 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20240501 20:00:54.173760 24327 tablet_bootstrap.cc:654] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:54.173956 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:54.169040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:37011 (local address 127.0.0.1:44844)
0501 20:00:54.169235 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:54.169250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:54.169331 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:54.171101 (+ 1770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:54.171104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:54.171122 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:54.171322 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:54.171331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.172797 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:54.172800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:54.173627 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:54.173637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.173742 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:54.173758 (+ 16us) client_negotiation.cc:769] Sending connection context
0501 20:00:54.173802 (+ 44us) client_negotiation.cc:240] Negotiation successful
0501 20:00:54.173856 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20240501 20:00:54.174250 24716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:54.169830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52164 (local address 127.8.168.67:34983)
0501 20:00:54.170103 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:54.170108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:54.170130 (+ 22us) server_negotiation.cc:407] Connection header received
0501 20:00:54.170216 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:54.170220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:54.170363 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:54.170501 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:54.170900 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.171942 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:54.173338 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.173849 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:54.173959 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:54.174028 (+ 69us) server_negotiation.cc:299] Negotiation successful
0501 20:00:54.174116 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":103,"threads_started":1}
I20240501 20:00:54.174970 24327 tablet_bootstrap.cc:492] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259: No bootstrap required, opened a new log
I20240501 20:00:54.175072 24327 ts_tablet_manager.cc:1367] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:00:54.175464 24327 raft_consensus.cc:348] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } } peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } }
I20240501 20:00:54.175578 24327 raft_consensus.cc:374] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:54.175626 24327 raft_consensus.cc:729] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ba60fb865a42c8bdf5b26b2e647259, State: Initialized, Role: FOLLOWER
I20240501 20:00:54.175761 24327 consensus_queue.cc:260] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } } peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } }
I20240501 20:00:54.176004 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:54.169212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44844 (local address 127.8.168.66:37011)
0501 20:00:54.170702 (+ 1490us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:54.170708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:54.170731 (+ 23us) server_negotiation.cc:407] Connection header received
0501 20:00:54.170826 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:54.170831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:54.170978 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:54.171103 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:54.171564 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.172638 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:54.175055 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.175602 (+ 547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:54.175720 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:54.175788 (+ 68us) server_negotiation.cc:299] Negotiation successful
0501 20:00:54.175873 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":125,"threads_started":1}
I20240501 20:00:54.176012 24327 ts_tablet_manager.cc:1398] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 20:00:54.176796 24639 tablet_service.cc:1467] Processing CreateTablet for tablet 9cdf00559c6c495bb53dabbb4ee633f4 (DEFAULT_TABLE table=test-table [id=794d65fe1fa24c3abe9a875c248c5ded]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:54.177716 24639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cdf00559c6c495bb53dabbb4ee633f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:54.178136 24507 tablet_service.cc:1467] Processing CreateTablet for tablet 9cdf00559c6c495bb53dabbb4ee633f4 (DEFAULT_TABLE table=test-table [id=794d65fe1fa24c3abe9a875c248c5ded]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:00:54.178807 24507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cdf00559c6c495bb53dabbb4ee633f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:54.183413 24718 tablet_bootstrap.cc:492] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572: Bootstrap starting.
I20240501 20:00:54.184160 24719 tablet_bootstrap.cc:492] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697: Bootstrap starting.
I20240501 20:00:54.185523 24718 tablet_bootstrap.cc:654] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:54.186080 24719 tablet_bootstrap.cc:654] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:54.186334 24718 log.cc:824] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:54.187186 24719 log.cc:824] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:54.188146 24718 tablet_bootstrap.cc:492] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572: No bootstrap required, opened a new log
I20240501 20:00:54.188318 24718 ts_tablet_manager.cc:1367] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:00:54.188989 24719 tablet_bootstrap.cc:492] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697: No bootstrap required, opened a new log
I20240501 20:00:54.189157 24719 ts_tablet_manager.cc:1367] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:00:54.191784 24718 raft_consensus.cc:348] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } } peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } }
I20240501 20:00:54.192000 24718 raft_consensus.cc:374] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:54.192056 24718 raft_consensus.cc:729] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa7730f481b4205b2813adf412b5572, State: Initialized, Role: FOLLOWER
I20240501 20:00:54.192065 24719 raft_consensus.cc:348] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } } peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } }
I20240501 20:00:54.192250 24719 raft_consensus.cc:374] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:54.192301 24719 raft_consensus.cc:729] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a823d5a9c1134ecfae1bed89bce5c697, State: Initialized, Role: FOLLOWER
I20240501 20:00:54.192495 24718 consensus_queue.cc:260] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } } peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } }
I20240501 20:00:54.192636 24719 consensus_queue.cc:260] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } } peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } }
I20240501 20:00:54.193254 24573 heartbeater.cc:503] Master 127.8.168.126:39523 was elected leader, sending a full tablet report...
I20240501 20:00:54.193387 24719 ts_tablet_manager.cc:1398] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:00:54.193387 24718 ts_tablet_manager.cc:1398] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20240501 20:00:54.193743 24705 heartbeater.cc:503] Master 127.8.168.126:39523 was elected leader, sending a full tablet report...
W20240501 20:00:54.214527 24574 tablet.cc:2369] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:00:54.233995 24723 raft_consensus.cc:482] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:54.234247 24723 raft_consensus.cc:504] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } } peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } }
I20240501 20:00:54.235538 24723 leader_election.cc:288] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5aa7730f481b4205b2813adf412b5572 (127.8.168.67:34983), b8ba60fb865a42c8bdf5b26b2e647259 (127.8.168.65:35715)
I20240501 20:00:54.244388 24575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:54.235770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:34983 (local address 127.8.168.66:46173)
0501 20:00:54.235961 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:54.235986 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:54.236075 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:54.240140 (+ 4065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:54.240143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:54.240164 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:54.240445 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:54.240453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.243399 (+ 2946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:54.243402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:54.244093 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:54.244102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.244188 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:54.244205 (+ 17us) client_negotiation.cc:769] Sending connection context
0501 20:00:54.244235 (+ 30us) client_negotiation.cc:240] Negotiation successful
0501 20:00:54.244280 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20240501 20:00:54.245244 24716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:54.235847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:46173 (local address 127.8.168.67:34983)
0501 20:00:54.239899 (+ 4052us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:54.239904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:54.239921 (+ 17us) server_negotiation.cc:407] Connection header received
0501 20:00:54.239973 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:54.239976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:54.240036 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:54.240121 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:54.242568 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.243281 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:54.244598 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.245023 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:54.245066 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:54.245110 (+ 44us) server_negotiation.cc:299] Negotiation successful
0501 20:00:54.245153 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3955}
I20240501 20:00:54.245731 24724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:54.236348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:35715 (local address 127.8.168.66:37537)
0501 20:00:54.242737 (+ 6389us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:54.242751 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 20:00:54.242893 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:00:54.243294 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:00:54.243297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:00:54.243315 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:00:54.243609 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:54.243617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.244601 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:00:54.244604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:00:54.245347 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:00:54.245361 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.245470 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:54.245490 (+ 20us) client_negotiation.cc:769] Sending connection context
0501 20:00:54.245538 (+ 48us) client_negotiation.cc:240] Negotiation successful
0501 20:00:54.245594 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6312,"thread_start_us":123,"threads_started":1}
I20240501 20:00:54.245781 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cdf00559c6c495bb53dabbb4ee633f4" candidate_uuid: "a823d5a9c1134ecfae1bed89bce5c697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa7730f481b4205b2813adf412b5572" is_pre_election: true
I20240501 20:00:54.246110 24659 raft_consensus.cc:2454] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a823d5a9c1134ecfae1bed89bce5c697 in term 0.
I20240501 20:00:54.246268 24443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:54.236698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:37537 (local address 127.8.168.65:35715)
0501 20:00:54.236855 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:54.236859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:54.242821 (+ 5962us) server_negotiation.cc:407] Connection header received
0501 20:00:54.243011 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:54.243015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:54.243084 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:54.243165 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:00:54.243741 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.244436 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:54.245468 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.245947 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:54.246072 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:54.246117 (+ 45us) server_negotiation.cc:299] Negotiation successful
0501 20:00:54.246159 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20240501 20:00:54.246568 24462 leader_election.cc:302] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa7730f481b4205b2813adf412b5572, a823d5a9c1134ecfae1bed89bce5c697; no voters:
I20240501 20:00:54.246807 24723 raft_consensus.cc:2781] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:54.246873 24723 raft_consensus.cc:482] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:54.246901 24723 raft_consensus.cc:3037] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:54.246917 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cdf00559c6c495bb53dabbb4ee633f4" candidate_uuid: "a823d5a9c1134ecfae1bed89bce5c697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" is_pre_election: true
I20240501 20:00:54.247200 24389 raft_consensus.cc:2454] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a823d5a9c1134ecfae1bed89bce5c697 in term 0.
I20240501 20:00:54.247988 24723 raft_consensus.cc:504] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } } peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } }
I20240501 20:00:54.248344 24723 leader_election.cc:288] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [CANDIDATE]: Term 1 election: Requested vote from peers 5aa7730f481b4205b2813adf412b5572 (127.8.168.67:34983), b8ba60fb865a42c8bdf5b26b2e647259 (127.8.168.65:35715)
I20240501 20:00:54.248735 24389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cdf00559c6c495bb53dabbb4ee633f4" candidate_uuid: "a823d5a9c1134ecfae1bed89bce5c697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8ba60fb865a42c8bdf5b26b2e647259"
I20240501 20:00:54.248764 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cdf00559c6c495bb53dabbb4ee633f4" candidate_uuid: "a823d5a9c1134ecfae1bed89bce5c697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa7730f481b4205b2813adf412b5572"
I20240501 20:00:54.248875 24659 raft_consensus.cc:3037] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:54.248868 24389 raft_consensus.cc:3037] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:54.249890 24659 raft_consensus.cc:2454] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a823d5a9c1134ecfae1bed89bce5c697 in term 1.
I20240501 20:00:54.250075 24389 raft_consensus.cc:2454] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a823d5a9c1134ecfae1bed89bce5c697 in term 1.
I20240501 20:00:54.250188 24462 leader_election.cc:302] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa7730f481b4205b2813adf412b5572, a823d5a9c1134ecfae1bed89bce5c697; no voters:
I20240501 20:00:54.250363 24723 raft_consensus.cc:2781] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:54.250603 24723 raft_consensus.cc:686] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [term 1 LEADER]: Becoming Leader. State: Replica: a823d5a9c1134ecfae1bed89bce5c697, State: Running, Role: LEADER
I20240501 20:00:54.250911 24723 consensus_queue.cc:237] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } } peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } }
I20240501 20:00:54.253935 24246 catalog_manager.cc:5484] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 reported cstate change: term changed from 0 to 1, leader changed from <none> to a823d5a9c1134ecfae1bed89bce5c697 (127.8.168.66). New cstate: current_term: 1 leader_uuid: "a823d5a9c1134ecfae1bed89bce5c697" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:54.275521 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:54.272838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44854 (local address 127.8.168.66:37011)
0501 20:00:54.272999 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0501 20:00:54.273002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:00:54.273016 (+ 14us) server_negotiation.cc:407] Connection header received
0501 20:00:54.273087 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:00:54.273089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:00:54.273143 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:00:54.273239 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:00:54.273497 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.274004 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:00:54.274587 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:00:54.274738 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:00:54.274825 (+ 87us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:00:54.275220 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:00:54.275310 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0501 20:00:54.275337 (+ 27us) server_negotiation.cc:299] Negotiation successful
0501 20:00:54.275381 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20240501 20:00:54.280710 24659 raft_consensus.cc:1261] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572 [term 1 FOLLOWER]: Refusing update from remote peer a823d5a9c1134ecfae1bed89bce5c697: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:00:54.280710 24389 raft_consensus.cc:1261] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Refusing update from remote peer a823d5a9c1134ecfae1bed89bce5c697: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:00:54.281356 24723 consensus_queue.cc:1035] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:54.281525 24725 consensus_queue.cc:1035] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:00:54.286383 24436 tablet.cc:2369] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:00:54.287400 24728 mvcc.cc:205] Tried to move back new op lower bound from 7022975607927218176 to 7022975607814250496. Current Snapshot: MvccSnapshot[applied={T|T < 7022975607927218176}]
I20240501 20:00:54.287920 24730 mvcc.cc:205] Tried to move back new op lower bound from 7022975607927218176 to 7022975607814250496. Current Snapshot: MvccSnapshot[applied={T|T < 7022975607927218176}]
I20240501 20:00:54.288067 24729 mvcc.cc:205] Tried to move back new op lower bound from 7022975607927218176 to 7022975607814250496. Current Snapshot: MvccSnapshot[applied={T|T < 7022975607927218176}]
I20240501 20:00:54.295728 24247 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56290:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:00:54.295887 24247 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56290:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:00:54.298164 24247 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: Sending DeleteTablet for 3 replicas of tablet 9cdf00559c6c495bb53dabbb4ee633f4
I20240501 20:00:54.298827 24369 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cdf00559c6c495bb53dabbb4ee633f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:00:54 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:36026
I20240501 20:00:54.298874 24507 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cdf00559c6c495bb53dabbb4ee633f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:00:54 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:44844
I20240501 20:00:54.298874 24639 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cdf00559c6c495bb53dabbb4ee633f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:00:54 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:52164
I20240501 20:00:54.299005 24444 tablet_replica.cc:330] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259: stopping tablet replica
I20240501 20:00:54.299116 24444 raft_consensus.cc:2229] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:54.299237 24444 raft_consensus.cc:2258] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:54.299626 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 24285
I20240501 20:00:54.300125 24739 tablet_replica.cc:330] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572: stopping tablet replica
I20240501 20:00:54.300395 24739 raft_consensus.cc:2229] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:54.300781 24739 raft_consensus.cc:2258] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:54.302042 24738 tablet_replica.cc:330] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697: stopping tablet replica
I20240501 20:00:54.302343 24738 raft_consensus.cc:2229] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:00:54.302713 24739 ts_tablet_manager.cc:1875] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:00:54.302855 24738 raft_consensus.cc:2258] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:54.304250 24738 ts_tablet_manager.cc:1875] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:00:54.305905 24739 ts_tablet_manager.cc:1888] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:00:54.306074 24739 log.cc:1196] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/9cdf00559c6c495bb53dabbb4ee633f4
I20240501 20:00:54.306574 24739 ts_tablet_manager.cc:1909] T 9cdf00559c6c495bb53dabbb4ee633f4 P 5aa7730f481b4205b2813adf412b5572: Deleting consensus metadata
I20240501 20:00:54.306710 24738 ts_tablet_manager.cc:1888] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:00:54.306865 24738 log.cc:1196] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/9cdf00559c6c495bb53dabbb4ee633f4
W20240501 20:00:54.307005 24234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.168.65:35715 (error 108)
I20240501 20:00:54.307243 24738 ts_tablet_manager.cc:1909] T 9cdf00559c6c495bb53dabbb4ee633f4 P a823d5a9c1134ecfae1bed89bce5c697: Deleting consensus metadata
I20240501 20:00:54.308005 24233 catalog_manager.cc:4830] TS 5aa7730f481b4205b2813adf412b5572 (127.8.168.67:34983): tablet 9cdf00559c6c495bb53dabbb4ee633f4 (table test-table [id=794d65fe1fa24c3abe9a875c248c5ded]) successfully deleted
I20240501 20:00:54.308157 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 24440
I20240501 20:00:54.308211 24231 catalog_manager.cc:4830] TS a823d5a9c1134ecfae1bed89bce5c697 (127.8.168.66:37011): tablet 9cdf00559c6c495bb53dabbb4ee633f4 (table test-table [id=794d65fe1fa24c3abe9a875c248c5ded]) successfully deleted
I20240501 20:00:54.310343 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:00:54.310084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:35715 (local address 127.0.0.1:36032)
0501 20:00:54.310210 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0501 20:00:54.310265 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:35715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20240501 20:00:54.310492 24234 catalog_manager.cc:4573] TS b8ba60fb865a42c8bdf5b26b2e647259 (127.8.168.65:35715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9cdf00559c6c495bb53dabbb4ee633f4: Network error: Client connection negotiation failed: client connection to 127.8.168.65:35715: connect: Connection refused (error 111)
I20240501 20:00:54.313606 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 24576
I20240501 20:00:54.318562 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 24214
I20240501 20:01:14.324311 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:39523
--webserver_interface=127.8.168.126
--webserver_port=41413
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:39523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:14.427727 24741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:14.427970 24741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:14.428025 24741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:14.431049 24741 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20240501 20:01:14.431098 24741 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20240501 20:01:14.431114 24741 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20240501 20:01:14.431128 24741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:14.431147 24741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:14.431172 24741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:14.431205 24741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:14.431233 24741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:14.435611 24741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:39523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:39523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=41413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:14.436491 24741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:14.437325 24741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:01:14.444954 24741 server_base.cc:1034] running on GCE node
W20240501 20:01:14.445478 24746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:14.445461 24749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:14.445871 24747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:14.446345 24741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:14.447204 24741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:14.448354 24741 hybrid_clock.cc:648] HybridClock initialized: now 1714593674448343 us; error 23 us; skew 500 ppm
May 01 20:01:14 dist-test-slave-tzct krb5kdc[23667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593674, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:14.450949 24741 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:01:14.451836 24741 webserver.cc:460] Webserver started at http://127.8.168.126:41413/ using document root <none> and password file <none>
I20240501 20:01:14.452345 24741 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:14.452399 24741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:14.455690 24741 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:14.457564 24756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:14.458542 24741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:14.458652 24741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "d43c9a64005e41efab8421ad002ddf76"
format_stamp: "Formatted at 2024-05-01 20:00:32 on dist-test-slave-tzct"
server_key: "6151a5281cdbf86d42ce39f35731b5ec"
server_key_iv: "e8baaa7c478998be38e53b28cf4612d3"
server_key_version: "encryptionkey@0"
I20240501 20:01:14.459054 24741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:14.488827 24741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:14.491782 24741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:14.491940 24741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:14.498453 24741 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:39523
I20240501 20:01:14.498654 24808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:39523 every 8 connection(s)
I20240501 20:01:14.499500 24741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:14.500114 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 24741
I20240501 20:01:14.500608 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:35715
--local_ip_for_outbound_sockets=127.8.168.65
--tserver_master_addrs=127.8.168.126:39523
--webserver_port=35677
--webserver_interface=127.8.168.65
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 20:01:14.503895 24809 sys_catalog.cc:263] Verifying existing consensus state
I20240501 20:01:14.505440 24809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: Bootstrap starting.
I20240501 20:01:14.521526 24809 log.cc:824] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:14.531987 24809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:14.532341 24809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: Bootstrap complete.
I20240501 20:01:14.535674 24809 raft_consensus.cc:348] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } }
I20240501 20:01:14.535987 24809 raft_consensus.cc:729] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43c9a64005e41efab8421ad002ddf76, State: Initialized, Role: FOLLOWER
I20240501 20:01:14.536437 24809 consensus_queue.cc:260] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } }
I20240501 20:01:14.536557 24809 raft_consensus.cc:388] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:14.536617 24809 raft_consensus.cc:482] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:14.536711 24809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:14.538484 24809 raft_consensus.cc:504] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } }
I20240501 20:01:14.538810 24809 leader_election.cc:302] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43c9a64005e41efab8421ad002ddf76; no voters:
I20240501 20:01:14.539108 24809 leader_election.cc:288] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [CANDIDATE]: Term 3 election: Requested vote from peers
I20240501 20:01:14.539233 24813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:14.539443 24813 raft_consensus.cc:686] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [term 3 LEADER]: Becoming Leader. State: Replica: d43c9a64005e41efab8421ad002ddf76, State: Running, Role: LEADER
I20240501 20:01:14.539795 24813 consensus_queue.cc:237] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } }
I20240501 20:01:14.540225 24809 sys_catalog.cc:564] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:14.541209 24815 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d43c9a64005e41efab8421ad002ddf76. Latest consensus state: current_term: 3 leader_uuid: "d43c9a64005e41efab8421ad002ddf76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } } }
I20240501 20:01:14.541357 24815 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:14.541595 24814 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d43c9a64005e41efab8421ad002ddf76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43c9a64005e41efab8421ad002ddf76" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39523 } } }
I20240501 20:01:14.541667 24814 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:14.545590 24828 catalog_manager.cc:1259] Loaded cluster ID: c53a43c37cb146a5a2f0ad2dd9359472
I20240501 20:01:14.545652 24828 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: loading cluster ID for follower catalog manager: success
I20240501 20:01:14.548058 24828 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d43c9a64005e41efab8421ad002ddf76: acquiring CA information for follower catalog manager: success
F20240501 20:01:14.549160 24828 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f356007adcd google::LogMessage::Fail() at ??:0
@ 0x7f356007eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f356007a7cc google::LogMessage::Flush() at ??:0
@ 0x7f356007bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f355e0e42be kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f3564f23420 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f3564efe57a kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f3564efeabe kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f3564ef764d kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f3564ef6961 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f3564f406de _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55a7538db75c std::function<>::operator()() at ??:0
@ 0x7f35614bb807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3561cf96db start_thread at ??:0
@ 0x7f355f23171f clone at ??:0
W20240501 20:01:14.622844 24811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:14.623090 24811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:14.623171 24811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:14.626343 24811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:14.626416 24811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:14.626518 24811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 20:01:14.630800 24811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:35715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=35677
--tserver_master_addrs=127.8.168.126:39523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:14.631716 24811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:14.632550 24811 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:14.641023 24850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:14.641299 24853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:14.641829 24851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:14.642144 24811 server_base.cc:1034] running on GCE node
I20240501 20:01:14.642529 24811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:14.643095 24811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:14.644275 24811 hybrid_clock.cc:648] HybridClock initialized: now 1714593674644253 us; error 39 us; skew 500 ppm
May 01 20:01:14 dist-test-slave-tzct krb5kdc[23667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593674, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:14.647491 24811 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 20:01:14.648672 24811 webserver.cc:460] Webserver started at http://127.8.168.65:35677/ using document root <none> and password file <none>
I20240501 20:01:14.649391 24811 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:14.649477 24811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:14.653889 24811 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240501 20:01:14.656338 24860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:14.657595 24811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:14.657728 24811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "b8ba60fb865a42c8bdf5b26b2e647259"
format_stamp: "Formatted at 2024-05-01 20:00:32 on dist-test-slave-tzct"
server_key: "5886a6aaaffc4a84ad0c7c9bd276ac9a"
server_key_iv: "e9c39f5e79127df8c784460527f53ff1"
server_key_version: "encryptionkey@0"
I20240501 20:01:14.658268 24811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:14.684582 24811 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:14.688241 24811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:14.688436 24811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:14.689165 24811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:14.690500 24867 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 20:01:14.694077 24811 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 20:01:14.694154 24811 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20240501 20:01:14.694218 24811 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 20:01:14.696256 24811 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 20:01:14.696316 24811 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240501 20:01:14.696576 24867 tablet_bootstrap.cc:492] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259: Bootstrap starting.
I20240501 20:01:14.701953 24867 log.cc:824] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:14.706506 24811 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:35715
I20240501 20:01:14.706773 24975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:35715 every 8 connection(s)
I20240501 20:01:14.707583 24811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 20:01:14.710274 24867 tablet_bootstrap.cc:492] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:14.710675 24867 tablet_bootstrap.cc:492] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259: Bootstrap complete.
I20240501 20:01:14.710985 24867 ts_tablet_manager.cc:1367] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20240501 20:01:14.712877 24867 raft_consensus.cc:348] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } } peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } }
I20240501 20:01:14.713181 24867 raft_consensus.cc:729] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ba60fb865a42c8bdf5b26b2e647259, State: Initialized, Role: FOLLOWER
I20240501 20:01:14.713589 24867 consensus_queue.cc:260] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } } peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } }
I20240501 20:01:14.714327 24867 ts_tablet_manager.cc:1398] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240501 20:01:14.717666 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 24811
I20240501 20:01:14.718070 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:37011
--local_ip_for_outbound_sockets=127.8.168.66
--tserver_master_addrs=127.8.168.126:39523
--webserver_port=37473
--webserver_interface=127.8.168.66
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:14.762419 24978 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:01:14.709471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:53935)
0501 20:01:14.709935 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:14.709978 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:14.710826 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:14.761822 (+ 50996us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":110,"threads_started":1}
W20240501 20:01:14.762964 24976 heartbeater.cc:644] Failed to heartbeat to 127.8.168.126:39523 (0 consecutive failures): Network error: Failed to ping master at 127.8.168.126:39523: Client connection negotiation failed: client connection to 127.8.168.126:39523: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:01:14.763787 24978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:14.763445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:44655)
0501 20:01:14.763625 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:14.763683 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20240501 20:01:14.764437 24978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:14.764205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:41105)
0501 20:01:14.764343 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:14.764383 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20240501 20:01:14.764577 24976 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240501 20:01:14.825049 24980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:14.825286 24980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:14.825363 24980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:14.828397 24980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:14.828462 24980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:14.828560 24980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 20:01:14.832571 24980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:37011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=37473
--tserver_master_addrs=127.8.168.126:39523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:14.833421 24980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:14.834223 24980 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:14.842084 24988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:14.842532 24985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:14.843029 24986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:14.843528 24980 server_base.cc:1034] running on GCE node
I20240501 20:01:14.843904 24980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:14.844394 24980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:14.845544 24980 hybrid_clock.cc:648] HybridClock initialized: now 1714593674845534 us; error 27 us; skew 500 ppm
May 01 20:01:14 dist-test-slave-tzct krb5kdc[23667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593674, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:14.848160 24980 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 20:01:14.849121 24980 webserver.cc:460] Webserver started at http://127.8.168.66:37473/ using document root <none> and password file <none>
I20240501 20:01:14.849633 24980 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:14.849714 24980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:14.853124 24980 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:14.855185 24995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:14.856217 24980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:14.856319 24980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "a823d5a9c1134ecfae1bed89bce5c697"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-tzct"
server_key: "5342e2f07c3fc12fc5f8584fe31b9d40"
server_key_iv: "16c93e65cb43f98f3178d77bdcb7cdc6"
server_key_version: "encryptionkey@0"
I20240501 20:01:14.856701 24980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:14.872627 24980 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:14.875440 24980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:14.875593 24980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:14.876139 24980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:14.876925 24980 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:14.876974 24980 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:14.877024 24980 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:14.877053 24980 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:14.885504 24980 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:37011
I20240501 20:01:14.885545 25108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:37011 every 8 connection(s)
I20240501 20:01:14.886442 24980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 20:01:14.889688 25111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:14.888238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:44861)
0501 20:01:14.888790 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:14.888893 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":94,"threads_started":1}
W20240501 20:01:14.890126 25109 heartbeater.cc:644] Failed to heartbeat to 127.8.168.126:39523 (0 consecutive failures): Network error: Failed to ping master at 127.8.168.126:39523: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
I20240501 20:01:14.890728 25111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:14.890495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:43001)
0501 20:01:14.890609 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:14.890657 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20240501 20:01:14.891372 25111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:14.891147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:53375)
0501 20:01:14.891275 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:14.891317 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20240501 20:01:14.891530 25109 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240501 20:01:14.893762 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 24980
I20240501 20:01:14.894140 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:34983
--local_ip_for_outbound_sockets=127.8.168.67
--tserver_master_addrs=127.8.168.126:39523
--webserver_port=41535
--webserver_interface=127.8.168.67
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:14.996304 25112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:14.996520 25112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:14.996572 25112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:14.999521 25112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:14.999576 25112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:14.999648 25112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 20:01:15.003561 25112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:34983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=41535
--tserver_master_addrs=127.8.168.126:39523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:15.004346 25112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:15.005179 25112 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:15.013402 25117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:15.013610 25120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:15.013756 25118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:15.014317 25112 server_base.cc:1034] running on GCE node
I20240501 20:01:15.014714 25112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:15.015319 25112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:15.016479 25112 hybrid_clock.cc:648] HybridClock initialized: now 1714593675016466 us; error 29 us; skew 500 ppm
May 01 20:01:15 dist-test-slave-tzct krb5kdc[23667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593675, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:15.019220 25112 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 20:01:15.020095 25112 webserver.cc:460] Webserver started at http://127.8.168.67:41535/ using document root <none> and password file <none>
I20240501 20:01:15.020601 25112 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:15.020658 25112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:15.024055 25112 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:15.025869 25127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:15.026854 25112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:01:15.026952 25112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "5aa7730f481b4205b2813adf412b5572"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-tzct"
server_key: "465d77d6537b7b7d8f82505574f85fb8"
server_key_iv: "5462dfd0b99d45605b53340821222c1a"
server_key_version: "encryptionkey@0"
I20240501 20:01:15.027346 25112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:15.040061 25112 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:15.042429 25112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:15.042577 25112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:15.043051 25112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:15.043783 25112 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:15.043830 25112 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:15.043880 25112 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:15.043907 25112 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:15.051983 25112 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:34983
I20240501 20:01:15.052037 25240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:34983 every 8 connection(s)
I20240501 20:01:15.052980 25112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 20:01:15.055938 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:15.054744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:35201)
0501 20:01:15.055156 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:15.055247 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":90,"threads_started":1}
W20240501 20:01:15.056365 25241 heartbeater.cc:644] Failed to heartbeat to 127.8.168.126:39523 (0 consecutive failures): Network error: Failed to ping master at 127.8.168.126:39523: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
I20240501 20:01:15.056983 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:15.056700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:58915)
0501 20:01:15.056886 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:15.056929 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20240501 20:01:15.057577 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:15.057342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:35015)
0501 20:01:15.057473 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:15.057520 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20240501 20:01:15.057735 25241 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240501 20:01:15.059417 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 25112
I20240501 20:01:15.765681 25245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:15.765247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:52561)
0501 20:01:15.765500 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:15.765560 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20240501 20:01:15.892570 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:15.892130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:42273)
0501 20:01:15.892396 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:15.892452 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20240501 20:01:16.058815 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:16.058337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:39653)
0501 20:01:16.058583 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:16.058654 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
I20240501 20:01:16.081656 25248 raft_consensus.cc:482] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:16.081868 25248 raft_consensus.cc:504] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } } peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } }
I20240501 20:01:16.082974 25248 leader_election.cc:288] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a823d5a9c1134ecfae1bed89bce5c697 (127.8.168.66:37011), 5aa7730f481b4205b2813adf412b5572 (127.8.168.67:34983)
May 01 20:01:16 dist-test-slave-tzct krb5kdc[23667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593674, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.66@KRBTEST.COM
May 01 20:01:16 dist-test-slave-tzct krb5kdc[23667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593674, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.67@KRBTEST.COM
I20240501 20:01:16.094173 25245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:16.083241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:37011 (local address 127.8.168.65:34619)
0501 20:01:16.083392 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:16.083408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:16.083525 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:16.085279 (+ 1754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:16.085284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:16.085762 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:16.086269 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:16.086284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:16.087539 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:16.087542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:16.087899 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:16.087906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:16.088001 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:16.088606 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:01:16.088630 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:16.090054 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:16.092215 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:16.092220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:16.092246 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:16.092456 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:16.092746 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:16.092749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:16.092751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:16.092833 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:16.093371 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:16.093376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:16.093577 (+ 201us) client_negotiation.cc:769] Sending connection context
0501 20:01:16.093744 (+ 167us) client_negotiation.cc:240] Negotiation successful
0501 20:01:16.094002 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20240501 20:01:16.094269 25251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:16.083483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:34619 (local address 127.8.168.66:37011)
0501 20:01:16.083820 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:16.083830 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:16.083869 (+ 39us) server_negotiation.cc:407] Connection header received
0501 20:01:16.084522 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:16.084527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:16.084912 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:16.086443 (+ 1531us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:16.086715 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:16.087423 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:16.088000 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:16.088254 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:16.090220 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:16.090258 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:16.090267 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:16.090301 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:16.092101 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:16.092560 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:16.092564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:16.092568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:16.092639 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:16.092914 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:16.092918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:16.092920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:16.093260 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:16.093384 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:16.093877 (+ 493us) server_negotiation.cc:299] Negotiation successful
0501 20:01:16.094111 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":68,"threads_started":1}
I20240501 20:01:16.094393 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:16.083363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:45257 (local address 127.8.168.67:34983)
0501 20:01:16.083749 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:16.083759 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:16.083799 (+ 40us) server_negotiation.cc:407] Connection header received
0501 20:01:16.084538 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:16.084546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:16.084907 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:16.085229 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:16.086445 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:16.087259 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:16.087871 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:16.088132 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:16.090482 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:16.090520 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:16.090532 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:16.090573 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:16.092480 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:16.092911 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:16.092915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:16.092919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:16.092988 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:16.093214 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:16.093218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:16.093220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:16.093535 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:16.093657 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:16.093988 (+ 331us) server_negotiation.cc:299] Negotiation successful
0501 20:01:16.094226 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":88,"threads_started":1}
I20240501 20:01:16.094321 25250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:16.083244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:34983 (local address 127.8.168.65:45257)
0501 20:01:16.083731 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:16.083745 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:16.083830 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:16.085281 (+ 1451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:16.085284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:16.085811 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:16.086270 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:16.086284 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:16.087386 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:16.087389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:16.087761 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:16.087768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:16.087924 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:16.088784 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:01:16.088797 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:16.090344 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:16.092592 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:16.092596 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:16.092598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:16.092809 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:16.093082 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:16.093085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:16.093087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:16.093139 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:16.093635 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:16.093638 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:16.093723 (+ 85us) client_negotiation.cc:769] Sending connection context
0501 20:01:16.093858 (+ 135us) client_negotiation.cc:240] Negotiation successful
0501 20:01:16.094101 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":255,"spinlock_wait_cycles":3968,"thread_start_us":63,"threads_started":1}
W20240501 20:01:16.096865 24863 leader_election.cc:341] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5aa7730f481b4205b2813adf412b5572 (127.8.168.67:34983): Not found: Tablet not found: 9cdf00559c6c495bb53dabbb4ee633f4
W20240501 20:01:16.097070 24861 leader_election.cc:341] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a823d5a9c1134ecfae1bed89bce5c697 (127.8.168.66:37011): Not found: Tablet not found: 9cdf00559c6c495bb53dabbb4ee633f4
I20240501 20:01:16.097119 24861 leader_election.cc:302] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8ba60fb865a42c8bdf5b26b2e647259; no voters: 5aa7730f481b4205b2813adf412b5572, a823d5a9c1134ecfae1bed89bce5c697
I20240501 20:01:16.097323 25248 raft_consensus.cc:2726] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:16.766834 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:16.766424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:46703)
0501 20:01:16.766676 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:16.766730 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
I20240501 20:01:16.893934 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:16.893367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:44373)
0501 20:01:16.893770 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:16.893836 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":72,"threads_started":1}
I20240501 20:01:17.060071 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:17.059609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:57239)
0501 20:01:17.059916 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:17.059975 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":72,"threads_started":1}
I20240501 20:01:17.768060 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:17.767601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:37731)
0501 20:01:17.767889 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:17.767952 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":91,"threads_started":1}
I20240501 20:01:17.895251 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:17.894718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:53537)
0501 20:01:17.895034 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:17.895093 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":76,"threads_started":1}
I20240501 20:01:18.061265 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:18.060851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:51765)
0501 20:01:18.061117 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:18.061168 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20240501 20:01:18.314811 25258 raft_consensus.cc:482] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:18.314960 25258 raft_consensus.cc:504] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } } peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } }
I20240501 20:01:18.315394 25258 leader_election.cc:288] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a823d5a9c1134ecfae1bed89bce5c697 (127.8.168.66:37011), 5aa7730f481b4205b2813adf412b5572 (127.8.168.67:34983)
W20240501 20:01:18.316167 24863 leader_election.cc:341] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5aa7730f481b4205b2813adf412b5572 (127.8.168.67:34983): Not found: Tablet not found: 9cdf00559c6c495bb53dabbb4ee633f4
W20240501 20:01:18.316275 24861 leader_election.cc:341] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a823d5a9c1134ecfae1bed89bce5c697 (127.8.168.66:37011): Not found: Tablet not found: 9cdf00559c6c495bb53dabbb4ee633f4
I20240501 20:01:18.316368 24861 leader_election.cc:302] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8ba60fb865a42c8bdf5b26b2e647259; no voters: 5aa7730f481b4205b2813adf412b5572, a823d5a9c1134ecfae1bed89bce5c697
I20240501 20:01:18.316494 25258 raft_consensus.cc:2726] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:18.769184 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:18.768792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:49597)
0501 20:01:18.769030 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:18.769083 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20240501 20:01:18.896559 25260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:18.896090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:56533)
0501 20:01:18.896396 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:18.896457 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":84,"threads_started":1}
I20240501 20:01:19.062547 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:19.062036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:39265)
0501 20:01:19.062344 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:19.062424 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":75,"threads_started":1}
I20240501 20:01:19.770602 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:19.770086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:49843)
0501 20:01:19.770437 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:19.770498 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":99,"threads_started":1}
I20240501 20:01:19.824002 25263 raft_consensus.cc:482] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:19.824143 25263 raft_consensus.cc:504] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } } peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } }
I20240501 20:01:19.824594 25263 leader_election.cc:288] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a823d5a9c1134ecfae1bed89bce5c697 (127.8.168.66:37011), 5aa7730f481b4205b2813adf412b5572 (127.8.168.67:34983)
W20240501 20:01:19.825407 24861 leader_election.cc:341] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a823d5a9c1134ecfae1bed89bce5c697 (127.8.168.66:37011): Not found: Tablet not found: 9cdf00559c6c495bb53dabbb4ee633f4
W20240501 20:01:19.825513 24863 leader_election.cc:341] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5aa7730f481b4205b2813adf412b5572 (127.8.168.67:34983): Not found: Tablet not found: 9cdf00559c6c495bb53dabbb4ee633f4
I20240501 20:01:19.825603 24863 leader_election.cc:302] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8ba60fb865a42c8bdf5b26b2e647259; no voters: 5aa7730f481b4205b2813adf412b5572, a823d5a9c1134ecfae1bed89bce5c697
I20240501 20:01:19.825739 25263 raft_consensus.cc:2726] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:19.897744 25264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:19.897303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:43525)
0501 20:01:19.897564 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:19.897620 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20240501 20:01:20.063766 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:20.063336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:32943)
0501 20:01:20.063578 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:20.063634 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20240501 20:01:20.771800 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:20.771402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:38987)
0501 20:01:20.771647 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:20.771700 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20240501 20:01:20.898983 25267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:20.898519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:40067)
0501 20:01:20.898819 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:20.898881 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
I20240501 20:01:21.065150 25268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:21.064640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:54407)
0501 20:01:21.064984 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:21.065047 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":172,"threads_started":1}
I20240501 20:01:21.773237 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:21.772701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:39059)
0501 20:01:21.773061 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:21.773127 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":83,"threads_started":1}
I20240501 20:01:21.900373 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:21.899844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:35891)
0501 20:01:21.900204 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:21.900271 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":84,"threads_started":1}
I20240501 20:01:22.066469 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:22.066000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:54955)
0501 20:01:22.066313 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:22.066368 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":75,"threads_started":1}
I20240501 20:01:22.686177 25272 raft_consensus.cc:482] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:22.686326 25272 raft_consensus.cc:504] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } } peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } }
I20240501 20:01:22.686738 25272 leader_election.cc:288] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a823d5a9c1134ecfae1bed89bce5c697 (127.8.168.66:37011), 5aa7730f481b4205b2813adf412b5572 (127.8.168.67:34983)
W20240501 20:01:22.687551 24863 leader_election.cc:341] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5aa7730f481b4205b2813adf412b5572 (127.8.168.67:34983): Not found: Tablet not found: 9cdf00559c6c495bb53dabbb4ee633f4
W20240501 20:01:22.687669 24861 leader_election.cc:341] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a823d5a9c1134ecfae1bed89bce5c697 (127.8.168.66:37011): Not found: Tablet not found: 9cdf00559c6c495bb53dabbb4ee633f4
I20240501 20:01:22.687727 24861 leader_election.cc:302] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8ba60fb865a42c8bdf5b26b2e647259; no voters: 5aa7730f481b4205b2813adf412b5572, a823d5a9c1134ecfae1bed89bce5c697
I20240501 20:01:22.687855 25272 raft_consensus.cc:2726] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:22.774602 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:22.774102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:54427)
0501 20:01:22.774408 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:22.774477 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":94,"threads_started":1}
I20240501 20:01:22.901629 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:22.901211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:56345)
0501 20:01:22.901479 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:22.901532 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
I20240501 20:01:23.067662 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:23.067233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:33469)
0501 20:01:23.067494 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:23.067556 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":70,"threads_started":1}
I20240501 20:01:23.775872 25276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:23.775430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:42371)
0501 20:01:23.775696 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:23.775749 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":74,"threads_started":1}
I20240501 20:01:23.902881 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:23.902406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:60275)
0501 20:01:23.902677 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:23.902735 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
I20240501 20:01:24.068823 25278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:24.068401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:53351)
0501 20:01:24.068644 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:24.068698 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20240501 20:01:24.745275 25279 raft_consensus.cc:482] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:24.745406 25279 raft_consensus.cc:504] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a823d5a9c1134ecfae1bed89bce5c697" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 37011 } } peers { permanent_uuid: "5aa7730f481b4205b2813adf412b5572" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 34983 } } peers { permanent_uuid: "b8ba60fb865a42c8bdf5b26b2e647259" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 35715 } }
I20240501 20:01:24.745803 25279 leader_election.cc:288] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a823d5a9c1134ecfae1bed89bce5c697 (127.8.168.66:37011), 5aa7730f481b4205b2813adf412b5572 (127.8.168.67:34983)
W20240501 20:01:24.746480 24863 leader_election.cc:341] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5aa7730f481b4205b2813adf412b5572 (127.8.168.67:34983): Not found: Tablet not found: 9cdf00559c6c495bb53dabbb4ee633f4
W20240501 20:01:24.746595 24861 leader_election.cc:341] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a823d5a9c1134ecfae1bed89bce5c697 (127.8.168.66:37011): Not found: Tablet not found: 9cdf00559c6c495bb53dabbb4ee633f4
I20240501 20:01:24.746656 24861 leader_election.cc:302] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8ba60fb865a42c8bdf5b26b2e647259; no voters: 5aa7730f481b4205b2813adf412b5572, a823d5a9c1134ecfae1bed89bce5c697
I20240501 20:01:24.746775 25279 raft_consensus.cc:2726] T 9cdf00559c6c495bb53dabbb4ee633f4 P b8ba60fb865a42c8bdf5b26b2e647259 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:24.777083 25280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:24.776666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:52249)
0501 20:01:24.776938 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:24.776987 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":115,"threads_started":1}
I20240501 20:01:24.904153 25281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:24.903728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:49875)
0501 20:01:24.903993 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:24.904051 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
I20240501 20:01:25.069965 25282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:25.069560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:33251)
0501 20:01:25.069807 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:25.069860 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20240501 20:01:25.778173 25283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:25.777725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:35997)
0501 20:01:25.778011 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:25.778072 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
I20240501 20:01:25.905303 25284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:25.904896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:54377)
0501 20:01:25.905145 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:25.905202 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
I20240501 20:01:26.071185 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:26.070706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:39029)
0501 20:01:26.071014 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:26.071078 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":86,"threads_started":1}
I20240501 20:01:26.779481 25286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:26.779050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:37725)
0501 20:01:26.779321 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:26.779379 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
I20240501 20:01:26.906504 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:26.906080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:37437)
0501 20:01:26.906337 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:26.906394 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
I20240501 20:01:27.072392 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:27.071955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:32899)
0501 20:01:27.072230 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:27.072280 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":106,"threads_started":1}
I20240501 20:01:27.780746 25289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:27.780257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:58673)
0501 20:01:27.780551 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:27.780621 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":99,"threads_started":1}
I20240501 20:01:27.907624 25290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:27.907172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:33369)
0501 20:01:27.907468 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:27.907523 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":73,"threads_started":1}
I20240501 20:01:28.073710 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:28.073250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:40695)
0501 20:01:28.073550 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:28.073611 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":102,"threads_started":1}
I20240501 20:01:28.782052 25292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:28.781625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:36187)
0501 20:01:28.781881 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:28.781941 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20240501 20:01:28.908836 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:28.908379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:60795)
0501 20:01:28.908643 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:28.908700 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20240501 20:01:29.074891 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:29.074467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:41577)
0501 20:01:29.074732 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:29.074790 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":87,"threads_started":1}
I20240501 20:01:29.783437 25295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:29.782981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.65:36347)
0501 20:01:29.783249 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:29.783332 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20240501 20:01:29.910068 25296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:29.909635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.66:49417)
0501 20:01:29.909907 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:29.909965 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":89,"threads_started":1}
I20240501 20:01:30.068115 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 24811
I20240501 20:01:30.073097 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 24980
I20240501 20:01:30.076997 25297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:30.076518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:39523 (local address 127.8.168.67:60779)
0501 20:01:30.076812 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:30.076877 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.126:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":67,"threads_started":1}
I20240501 20:01:30.080125 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 25112
2024-05-01T20:01:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58650 ms)
[----------] 30 tests from SecurityITest (336198 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:30 dist-test-slave-tzct krb5kdc[25300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:30 dist-test-slave-tzct krb5kdc[25300](info): set up 2 sockets
May 01 20:01:30 dist-test-slave-tzct krb5kdc[25300](info): commencing operation
krb5kdc: starting...
W20240501 20:01:30.988544 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.887s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:31 dist-test-slave-tzct krb5kdc[25300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 20:01:31.099022 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:35575
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:35575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:31.202374 25313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:31.202636 25313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:31.202708 25313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:31.205822 25313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:31.205899 25313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:31.205931 25313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:31.205956 25313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:31.205988 25313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:31.210177 25313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:35575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:35575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:31.211094 25313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:31.211926 25313 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:31.219060 25319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:31.219358 25318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:31.219864 25321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:31.220321 25313 server_base.cc:1034] running on GCE node
I20240501 20:01:31.220810 25313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:31.221719 25313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:31.222864 25313 hybrid_clock.cc:648] HybridClock initialized: now 1714593691222844 us; error 33 us; skew 500 ppm
May 01 20:01:31 dist-test-slave-tzct krb5kdc[25300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593691, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:31.225435 25313 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:01:31.226442 25313 webserver.cc:460] Webserver started at http://127.8.168.126:44197/ using document root <none> and password file <none>
I20240501 20:01:31.226995 25313 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:31.227059 25313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:31.227259 25313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:31.229125 25313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "c20efebf24614e26a3fcc3e6cc4e49d3"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-tzct"
server_key: "23c8cf6a79d86b0f8d40c7e5cb4387ff"
server_key_iv: "1db8cd55a4a7cf6879da4fae4264fc3e"
server_key_version: "encryptionkey@0"
I20240501 20:01:31.229624 25313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "c20efebf24614e26a3fcc3e6cc4e49d3"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-tzct"
server_key: "23c8cf6a79d86b0f8d40c7e5cb4387ff"
server_key_iv: "1db8cd55a4a7cf6879da4fae4264fc3e"
server_key_version: "encryptionkey@0"
I20240501 20:01:31.233012 25313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:01:31.235114 25328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.236124 25313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:31.236248 25313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "c20efebf24614e26a3fcc3e6cc4e49d3"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-tzct"
server_key: "23c8cf6a79d86b0f8d40c7e5cb4387ff"
server_key_iv: "1db8cd55a4a7cf6879da4fae4264fc3e"
server_key_version: "encryptionkey@0"
I20240501 20:01:31.236369 25313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:31.252144 25313 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:31.255141 25313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:31.255308 25313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:31.261554 25313 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:35575
I20240501 20:01:31.261569 25380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:35575 every 8 connection(s)
I20240501 20:01:31.262487 25313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:31.264160 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 25313
I20240501 20:01:31.264277 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:31.264520 8865 external_mini_cluster.cc:1428] Setting key 09e2e54053f24125a76aedcfe169add5
I20240501 20:01:31.265355 25381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:31.271085 25381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3: Bootstrap starting.
May 01 20:01:31 dist-test-slave-tzct krb5kdc[25300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:01:31.273185 25381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:31.274019 25381 log.cc:824] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:31.276010 25381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3: No bootstrap required, opened a new log
I20240501 20:01:31.277253 25384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:31.265710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.8.168.126:35575)
0501 20:01:31.266138 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:31.266149 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:31.266187 (+ 38us) server_negotiation.cc:407] Connection header received
0501 20:01:31.266938 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:31.266960 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:31.267294 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:31.267685 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:31.268417 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.269550 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:31.270246 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.270515 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:31.272479 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:31.272518 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:31.272531 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:31.272571 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:31.274575 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:31.275052 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:31.275057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:31.275062 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:31.275148 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:31.275367 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:31.275370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:31.275371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:31.275805 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:31.275987 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:31.276247 (+ 260us) server_negotiation.cc:299] Negotiation successful
0501 20:01:31.276518 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":144,"threads_started":1}
I20240501 20:01:31.279434 25381 raft_consensus.cc:348] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20efebf24614e26a3fcc3e6cc4e49d3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 35575 } }
I20240501 20:01:31.279608 25381 raft_consensus.cc:374] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:31.279649 25381 raft_consensus.cc:729] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20efebf24614e26a3fcc3e6cc4e49d3, State: Initialized, Role: FOLLOWER
I20240501 20:01:31.280014 25381 consensus_queue.cc:260] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20efebf24614e26a3fcc3e6cc4e49d3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 35575 } }
I20240501 20:01:31.280125 25381 raft_consensus.cc:388] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:31.280167 25381 raft_consensus.cc:482] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:31.280241 25381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:31.281041 25381 raft_consensus.cc:504] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20efebf24614e26a3fcc3e6cc4e49d3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 35575 } }
I20240501 20:01:31.281322 25381 leader_election.cc:302] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c20efebf24614e26a3fcc3e6cc4e49d3; no voters:
I20240501 20:01:31.281549 25381 leader_election.cc:288] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:31.281666 25386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:31.281891 25386 raft_consensus.cc:686] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [term 1 LEADER]: Becoming Leader. State: Replica: c20efebf24614e26a3fcc3e6cc4e49d3, State: Running, Role: LEADER
I20240501 20:01:31.282301 25386 consensus_queue.cc:237] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20efebf24614e26a3fcc3e6cc4e49d3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 35575 } }
I20240501 20:01:31.282617 25381 sys_catalog.cc:564] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:31.283671 25387 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c20efebf24614e26a3fcc3e6cc4e49d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20efebf24614e26a3fcc3e6cc4e49d3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 35575 } } }
I20240501 20:01:31.283800 25387 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:31.284049 25388 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c20efebf24614e26a3fcc3e6cc4e49d3. Latest consensus state: current_term: 1 leader_uuid: "c20efebf24614e26a3fcc3e6cc4e49d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20efebf24614e26a3fcc3e6cc4e49d3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 35575 } } }
I20240501 20:01:31.284214 25388 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:31.284119 25395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:31.286761 25395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:31.291371 25395 catalog_manager.cc:1347] Generated new cluster ID: 4b2f3a9fdbde46debfa32464b83c82ee
I20240501 20:01:31.291446 25395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:31.327469 25395 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:31.328424 25395 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:31.335392 25395 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3: Generated new TSK 0
I20240501 20:01:31.336064 25395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
I20240501 20:01:31.365844 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:35575
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:31.478341 25407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:31.478567 25407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:31.478632 25407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:31.481683 25407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:31.481742 25407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:31.481842 25407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 20:01:31.486030 25407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:31.487054 25407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:31.487936 25407 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:31.496328 25415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:31.496448 25412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:31.496791 25413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:31.497381 25407 server_base.cc:1034] running on GCE node
I20240501 20:01:31.497761 25407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:31.498343 25407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:31.499516 25407 hybrid_clock.cc:648] HybridClock initialized: now 1714593691499501 us; error 29 us; skew 500 ppm
May 01 20:01:31 dist-test-slave-tzct krb5kdc[25300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593691, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:31.502295 25407 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 20:01:31.503264 25407 webserver.cc:460] Webserver started at http://127.8.168.65:44635/ using document root <none> and password file <none>
I20240501 20:01:31.503788 25407 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:31.503845 25407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:31.504021 25407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:31.505832 25407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "6d83eb2d74b54f778234af973d723126"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-tzct"
server_key: "7538e0d18ef081fb22d5d1f364204544"
server_key_iv: "929461a395326a27a9de3f5f416b9a58"
server_key_version: "encryptionkey@0"
I20240501 20:01:31.506290 25407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "6d83eb2d74b54f778234af973d723126"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-tzct"
server_key: "7538e0d18ef081fb22d5d1f364204544"
server_key_iv: "929461a395326a27a9de3f5f416b9a58"
server_key_version: "encryptionkey@0"
I20240501 20:01:31.509605 25407 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:01:31.511742 25422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.512782 25407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:31.512933 25407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "6d83eb2d74b54f778234af973d723126"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-tzct"
server_key: "7538e0d18ef081fb22d5d1f364204544"
server_key_iv: "929461a395326a27a9de3f5f416b9a58"
server_key_version: "encryptionkey@0"
I20240501 20:01:31.513034 25407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:31.536322 25407 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:31.539220 25407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:31.539456 25407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:31.540096 25407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:31.541092 25407 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:31.541153 25407 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.541208 25407 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:31.541240 25407 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.549773 25407 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:37745
I20240501 20:01:31.549805 25535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:37745 every 8 connection(s)
I20240501 20:01:31.550689 25407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 20:01:31.551285 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 25407
I20240501 20:01:31.551429 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 20:01:31.551671 8865 external_mini_cluster.cc:1428] Setting key 5f12cafba4daabd108fffbd94e0a6f6e
May 01 20:01:31 dist-test-slave-tzct krb5kdc[25300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593691, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.168.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.66@KRBTEST.COM" created.
I20240501 20:01:31.563742 25384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:31.552369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:34841 (local address 127.8.168.126:35575)
0501 20:01:31.552501 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:31.552505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:31.553536 (+ 1031us) server_negotiation.cc:407] Connection header received
0501 20:01:31.554278 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:31.554281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:31.554355 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:31.554444 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:31.556272 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.557076 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:31.557952 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.558100 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:31.560601 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:31.560620 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:31.560623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:31.560652 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:31.562026 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:31.562458 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:31.562461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:31.562462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:31.562500 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:31.562735 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:31.562737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:31.562739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:31.562880 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:31.562955 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:31.563439 (+ 484us) server_negotiation.cc:299] Negotiation successful
0501 20:01:31.563575 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240501 20:01:31.564378 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:31.552659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:35575 (local address 127.8.168.65:34841)
0501 20:01:31.553374 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:31.553426 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:31.554088 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:31.554621 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:31.554632 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:31.555146 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:31.556108 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:31.556121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.557201 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:31.557205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:31.557823 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:31.557835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.558071 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:31.558683 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:01:31.558718 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:31.560440 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:31.562141 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:31.562145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:31.562153 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:31.562369 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:31.562577 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:31.562579 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:31.562580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:31.562662 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:31.562956 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:31.562960 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:31.563177 (+ 217us) client_negotiation.cc:769] Sending connection context
0501 20:01:31.563357 (+ 180us) client_negotiation.cc:240] Negotiation successful
0501 20:01:31.563603 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":147,"threads_started":1}
I20240501 20:01:31.565513 25536 heartbeater.cc:347] Connected to a master server at 127.8.168.126:35575
I20240501 20:01:31.565730 25536 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:31.566303 25536 heartbeater.cc:511] Master 127.8.168.126:35575 requested a full tablet report, sending...
I20240501 20:01:31.567749 25345 ts_manager.cc:194] Registered new tserver with Master: 6d83eb2d74b54f778234af973d723126 (127.8.168.65:37745)
I20240501 20:01:31.568948 25345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:34841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
I20240501 20:01:31.577562 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:35575
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:31.680334 25541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:31.680555 25541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:31.680610 25541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:31.683672 25541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:31.683728 25541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:31.683804 25541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 20:01:31.687858 25541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:31.688740 25541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:31.689574 25541 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:31.697779 25549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:31.697831 25546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:31.698349 25547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:31.698803 25541 server_base.cc:1034] running on GCE node
I20240501 20:01:31.699172 25541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:31.699717 25541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:31.700856 25541 hybrid_clock.cc:648] HybridClock initialized: now 1714593691700839 us; error 34 us; skew 500 ppm
May 01 20:01:31 dist-test-slave-tzct krb5kdc[25300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593691, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:31.703370 25541 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 20:01:31.704257 25541 webserver.cc:460] Webserver started at http://127.8.168.66:37519/ using document root <none> and password file <none>
I20240501 20:01:31.704811 25541 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:31.704872 25541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:31.705046 25541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:31.706743 25541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "42f8c5e5b7eb41189fb255d61ef1a12c"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-tzct"
server_key: "2b0f5d152f0a730c5c82f5edf206cece"
server_key_iv: "92f11f3ce39b2358113cecfc7a6b5dec"
server_key_version: "encryptionkey@0"
I20240501 20:01:31.707185 25541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "42f8c5e5b7eb41189fb255d61ef1a12c"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-tzct"
server_key: "2b0f5d152f0a730c5c82f5edf206cece"
server_key_iv: "92f11f3ce39b2358113cecfc7a6b5dec"
server_key_version: "encryptionkey@0"
I20240501 20:01:31.710778 25541 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 20:01:31.713074 25556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.714080 25541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:31.714182 25541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "42f8c5e5b7eb41189fb255d61ef1a12c"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-tzct"
server_key: "2b0f5d152f0a730c5c82f5edf206cece"
server_key_iv: "92f11f3ce39b2358113cecfc7a6b5dec"
server_key_version: "encryptionkey@0"
I20240501 20:01:31.714270 25541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:31.741155 25541 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:31.744004 25541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:31.744172 25541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:31.744700 25541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:31.745528 25541 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:31.745577 25541 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.745630 25541 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:31.745657 25541 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.753994 25541 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:43961
I20240501 20:01:31.754154 25669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:43961 every 8 connection(s)
I20240501 20:01:31.754932 25541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
May 01 20:01:31 dist-test-slave-tzct krb5kdc[25300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593691, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:01:31.762902 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 25541
I20240501 20:01:31.763046 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 20:01:31.763324 8865 external_mini_cluster.cc:1428] Setting key 0125773f0520592676a8dfc7d82ce4e4
I20240501 20:01:31.766539 25384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:31.756562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:57517 (local address 127.8.168.126:35575)
0501 20:01:31.756692 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:31.756696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:31.757447 (+ 751us) server_negotiation.cc:407] Connection header received
0501 20:01:31.758193 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:31.758196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:31.758241 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:31.758312 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:31.759822 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.760325 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:31.760935 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.761063 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:31.763233 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:31.763253 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:31.763255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:31.763279 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:31.764604 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:31.765134 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:31.765136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:31.765138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:31.765177 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:31.765501 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:31.765503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:31.765504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:31.765634 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:31.765699 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:31.766312 (+ 613us) server_negotiation.cc:299] Negotiation successful
0501 20:01:31.766424 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 20:01:31.767396 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:31.756828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:35575 (local address 127.8.168.66:57517)
0501 20:01:31.757268 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:31.757335 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:31.758019 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:31.758452 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:31.758460 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:31.758817 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:31.759712 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:31.759725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.760412 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:31.760415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:31.760845 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:31.760852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.761010 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:31.761545 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:01:31.761572 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:31.763081 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:31.764706 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:31.764712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:31.764739 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:31.765050 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:31.765288 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:31.765292 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:31.765294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:31.765415 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:31.765723 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:31.765728 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:31.766014 (+ 286us) client_negotiation.cc:769] Sending connection context
0501 20:01:31.766219 (+ 205us) client_negotiation.cc:240] Negotiation successful
0501 20:01:31.766500 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":98,"threads_started":1}
I20240501 20:01:31.768426 25670 heartbeater.cc:347] Connected to a master server at 127.8.168.126:35575
I20240501 20:01:31.768638 25670 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:31.769115 25670 heartbeater.cc:511] Master 127.8.168.126:35575 requested a full tablet report, sending...
I20240501 20:01:31.770068 25345 ts_manager.cc:194] Registered new tserver with Master: 42f8c5e5b7eb41189fb255d61ef1a12c (127.8.168.66:43961)
I20240501 20:01:31.770603 25345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:57517
WARNING: no policy specified for kudu/127.8.168.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
I20240501 20:01:31.787509 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:35575
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:31.889902 25675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:31.890106 25675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:31.890164 25675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:31.893190 25675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:31.893249 25675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:31.893334 25675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 20:01:31.897351 25675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:31.898160 25675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:31.898947 25675 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:31.906855 25683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:31.907016 25680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:31.907459 25681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:31.907814 25675 server_base.cc:1034] running on GCE node
I20240501 20:01:31.908170 25675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:31.908753 25675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:31.909926 25675 hybrid_clock.cc:648] HybridClock initialized: now 1714593691909914 us; error 25 us; skew 500 ppm
May 01 20:01:31 dist-test-slave-tzct krb5kdc[25300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593691, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:31.912449 25675 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 20:01:31.913344 25675 webserver.cc:460] Webserver started at http://127.8.168.67:46753/ using document root <none> and password file <none>
I20240501 20:01:31.913847 25675 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:31.913911 25675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:31.914098 25675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:31.915848 25675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "aec0ee4a13094c26b3c2e4dd6ef93a53"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-tzct"
server_key: "6e74d9c8afa9db27542dcf584d2d020d"
server_key_iv: "ff4cc6e3b9ab19bf61bc8d708a6b44c2"
server_key_version: "encryptionkey@0"
I20240501 20:01:31.916307 25675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "aec0ee4a13094c26b3c2e4dd6ef93a53"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-tzct"
server_key: "6e74d9c8afa9db27542dcf584d2d020d"
server_key_iv: "ff4cc6e3b9ab19bf61bc8d708a6b44c2"
server_key_version: "encryptionkey@0"
I20240501 20:01:31.919517 25675 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 20:01:31.921670 25690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.922639 25675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:31.922741 25675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "aec0ee4a13094c26b3c2e4dd6ef93a53"
format_stamp: "Formatted at 2024-05-01 20:01:31 on dist-test-slave-tzct"
server_key: "6e74d9c8afa9db27542dcf584d2d020d"
server_key_iv: "ff4cc6e3b9ab19bf61bc8d708a6b44c2"
server_key_version: "encryptionkey@0"
I20240501 20:01:31.922824 25675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:31.937192 25675 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:31.940220 25675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:31.940382 25675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:31.940970 25675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:31.941743 25675 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:31.941792 25675 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.941843 25675 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:31.941871 25675 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:31.950392 25675 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:40021
I20240501 20:01:31.950431 25803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:40021 every 8 connection(s)
I20240501 20:01:31.951282 25675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 20:01:31.952651 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 25675
I20240501 20:01:31.952805 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 20:01:31.953125 8865 external_mini_cluster.cc:1428] Setting key 445ef3e28583f10d7e07e57267072827
May 01 20:01:31 dist-test-slave-tzct krb5kdc[25300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593691, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:01:31.962592 25384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:31.953120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:48283 (local address 127.8.168.126:35575)
0501 20:01:31.953284 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:31.953287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:31.954000 (+ 713us) server_negotiation.cc:407] Connection header received
0501 20:01:31.954814 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:31.954818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:31.954875 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:31.954944 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:31.956515 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.957044 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:31.957649 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.957828 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:31.959862 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:31.959879 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:31.959882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:31.959909 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:31.961062 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:31.961514 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:31.961517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:31.961519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:31.961568 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:31.961779 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:31.961781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:31.961782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:31.961917 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:31.961986 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:31.962376 (+ 390us) server_negotiation.cc:299] Negotiation successful
0501 20:01:31.962473 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20240501 20:01:31.963171 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:31.953433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:35575 (local address 127.8.168.67:48283)
0501 20:01:31.953856 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:31.953907 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:31.954613 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:31.955109 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:31.955117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:31.955474 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:31.956375 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:31.956389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.957151 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:31.957154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:31.957538 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:31.957545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.957739 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:31.958329 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:01:31.958354 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:31.959720 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:31.961157 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:31.961162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:31.961169 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:31.961382 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:31.961646 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:31.961648 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:31.961649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:31.961718 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:31.961985 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:31.961988 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:31.962188 (+ 200us) client_negotiation.cc:769] Sending connection context
0501 20:01:31.962339 (+ 151us) client_negotiation.cc:240] Negotiation successful
0501 20:01:31.962546 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":94,"threads_started":1}
I20240501 20:01:31.964141 25804 heartbeater.cc:347] Connected to a master server at 127.8.168.126:35575
I20240501 20:01:31.964337 25804 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:31.964814 25804 heartbeater.cc:511] Master 127.8.168.126:35575 requested a full tablet report, sending...
I20240501 20:01:31.965754 25345 ts_manager.cc:194] Registered new tserver with Master: aec0ee4a13094c26b3c2e4dd6ef93a53 (127.8.168.67:40021)
I20240501 20:01:31.966316 25345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:48283
I20240501 20:01:31.966720 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 20:01:31.973758 25384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:31.968440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47808 (local address 127.8.168.126:35575)
0501 20:01:31.968544 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:31.968547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:31.968662 (+ 115us) server_negotiation.cc:407] Connection header received
0501 20:01:31.968781 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:31.968783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:31.968818 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:31.968891 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:31.969583 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.970027 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:31.970571 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.970681 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:31.971592 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:31.971604 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:31.971607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:31.971621 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:31.972689 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:31.973040 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:31.973042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:31.973043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:31.973079 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:31.973213 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:31.973215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:31.973216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:31.973362 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:31.973454 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:31.973571 (+ 117us) server_negotiation.cc:299] Negotiation successful
0501 20:01:31.973650 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240501 20:01:31.975971 25345 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:01:31.978401 25345 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:01:31.990263 25817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:31.985669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:37745 (local address 127.0.0.1:36658)
0501 20:01:31.986478 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:31.986490 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:31.986572 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:31.987410 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:31.987416 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:31.987454 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:31.987757 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:31.987765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.988860 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:31.988863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:31.989736 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:31.989745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.989845 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:31.989894 (+ 49us) client_negotiation.cc:769] Sending connection context
0501 20:01:31.989981 (+ 87us) client_negotiation.cc:240] Negotiation successful
0501 20:01:31.990061 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":88,"threads_started":1}
I20240501 20:01:31.990828 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:31.986238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:40021 (local address 127.0.0.1:38772)
0501 20:01:31.987621 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:31.987634 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:31.987717 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:31.988127 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:31.988130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:31.988148 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:31.988358 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:31.988365 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.989546 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:31.989549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:31.990285 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:31.990295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.990590 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:31.990608 (+ 18us) client_negotiation.cc:769] Sending connection context
0501 20:01:31.990659 (+ 51us) client_negotiation.cc:240] Negotiation successful
0501 20:01:31.990713 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1320,"thread_start_us":102,"threads_started":1}
I20240501 20:01:31.991092 25818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:31.986211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44820 (local address 127.8.168.66:43961)
0501 20:01:31.986975 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:31.986982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:31.987005 (+ 23us) server_negotiation.cc:407] Connection header received
0501 20:01:31.987093 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:31.987097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:31.987257 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:31.987390 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:31.987883 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.988920 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:31.990184 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.990660 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:31.990775 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:31.990857 (+ 82us) server_negotiation.cc:299] Negotiation successful
0501 20:01:31.990951 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":153,"threads_started":1}
I20240501 20:01:31.991201 25820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:31.986633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36658 (local address 127.8.168.65:37745)
0501 20:01:31.986995 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:31.987001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:31.987024 (+ 23us) server_negotiation.cc:407] Connection header received
0501 20:01:31.987111 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:31.987117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:31.987266 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:31.987400 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:31.987876 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.988763 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:31.990146 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.990852 (+ 706us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:31.990947 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:31.991005 (+ 58us) server_negotiation.cc:299] Negotiation successful
0501 20:01:31.991073 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":138,"threads_started":1}
I20240501 20:01:31.991362 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:31.985583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:43961 (local address 127.0.0.1:44820)
0501 20:01:31.985878 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:31.985908 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:31.986005 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:31.987411 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:31.987417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:31.987476 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:31.987754 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:31.987762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.989022 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:31.989025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:31.989817 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:31.989834 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.989917 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:31.989928 (+ 11us) client_negotiation.cc:769] Sending connection context
0501 20:01:31.989978 (+ 50us) client_negotiation.cc:240] Negotiation successful
0501 20:01:31.990061 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"spinlock_wait_cycles":61184,"thread_start_us":87,"threads_started":1}
I20240501 20:01:31.991493 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:31.986409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38772 (local address 127.8.168.67:40021)
0501 20:01:31.987035 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:31.987040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:31.987729 (+ 689us) server_negotiation.cc:407] Connection header received
0501 20:01:31.987827 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:31.987832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:31.987998 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:31.988131 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:31.988474 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.989436 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:31.990611 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:31.991101 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:31.991203 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:31.991267 (+ 64us) server_negotiation.cc:299] Negotiation successful
0501 20:01:31.991345 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":134,"threads_started":1}
I20240501 20:01:31.993512 25470 tablet_service.cc:1467] Processing CreateTablet for tablet d718822eadd047d4916e00d3ac94a99f (DEFAULT_TABLE table=test-table [id=ac7e3ebb44e349129686fe2d938aab71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:31.993513 25738 tablet_service.cc:1467] Processing CreateTablet for tablet d718822eadd047d4916e00d3ac94a99f (DEFAULT_TABLE table=test-table [id=ac7e3ebb44e349129686fe2d938aab71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:31.994280 25738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d718822eadd047d4916e00d3ac94a99f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:31.994280 25470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d718822eadd047d4916e00d3ac94a99f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:31.998100 25603 tablet_service.cc:1467] Processing CreateTablet for tablet d718822eadd047d4916e00d3ac94a99f (DEFAULT_TABLE table=test-table [id=ac7e3ebb44e349129686fe2d938aab71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:31.998960 25603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d718822eadd047d4916e00d3ac94a99f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:31.999346 25821 tablet_bootstrap.cc:492] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126: Bootstrap starting.
I20240501 20:01:32.000239 25822 tablet_bootstrap.cc:492] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53: Bootstrap starting.
I20240501 20:01:32.001317 25821 tablet_bootstrap.cc:654] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:32.001968 25821 log.cc:824] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:32.002127 25822 tablet_bootstrap.cc:654] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:32.002867 25822 log.cc:824] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:32.003695 25821 tablet_bootstrap.cc:492] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126: No bootstrap required, opened a new log
I20240501 20:01:32.003870 25821 ts_tablet_manager.cc:1367] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:01:32.004639 25822 tablet_bootstrap.cc:492] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53: No bootstrap required, opened a new log
I20240501 20:01:32.004791 25825 tablet_bootstrap.cc:492] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c: Bootstrap starting.
I20240501 20:01:32.004823 25822 ts_tablet_manager.cc:1367] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:01:32.006430 25825 tablet_bootstrap.cc:654] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:32.006999 25825 log.cc:824] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:32.007211 25821 raft_consensus.cc:348] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f8c5e5b7eb41189fb255d61ef1a12c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43961 } } peers { permanent_uuid: "6d83eb2d74b54f778234af973d723126" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37745 } } peers { permanent_uuid: "aec0ee4a13094c26b3c2e4dd6ef93a53" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40021 } }
I20240501 20:01:32.007424 25821 raft_consensus.cc:374] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:32.007506 25821 raft_consensus.cc:729] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d83eb2d74b54f778234af973d723126, State: Initialized, Role: FOLLOWER
I20240501 20:01:32.007937 25821 consensus_queue.cc:260] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f8c5e5b7eb41189fb255d61ef1a12c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43961 } } peers { permanent_uuid: "6d83eb2d74b54f778234af973d723126" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37745 } } peers { permanent_uuid: "aec0ee4a13094c26b3c2e4dd6ef93a53" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40021 } }
I20240501 20:01:32.007987 25822 raft_consensus.cc:348] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f8c5e5b7eb41189fb255d61ef1a12c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43961 } } peers { permanent_uuid: "6d83eb2d74b54f778234af973d723126" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37745 } } peers { permanent_uuid: "aec0ee4a13094c26b3c2e4dd6ef93a53" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40021 } }
I20240501 20:01:32.008591 25536 heartbeater.cc:503] Master 127.8.168.126:35575 was elected leader, sending a full tablet report...
I20240501 20:01:32.008791 25822 raft_consensus.cc:374] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:32.008819 25821 ts_tablet_manager.cc:1398] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 20:01:32.008898 25822 raft_consensus.cc:729] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec0ee4a13094c26b3c2e4dd6ef93a53, State: Initialized, Role: FOLLOWER
I20240501 20:01:32.009272 25822 consensus_queue.cc:260] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f8c5e5b7eb41189fb255d61ef1a12c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43961 } } peers { permanent_uuid: "6d83eb2d74b54f778234af973d723126" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37745 } } peers { permanent_uuid: "aec0ee4a13094c26b3c2e4dd6ef93a53" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40021 } }
I20240501 20:01:32.009828 25804 heartbeater.cc:503] Master 127.8.168.126:35575 was elected leader, sending a full tablet report...
I20240501 20:01:32.009874 25825 tablet_bootstrap.cc:492] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c: No bootstrap required, opened a new log
I20240501 20:01:32.010023 25825 ts_tablet_manager.cc:1367] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:01:32.010022 25822 ts_tablet_manager.cc:1398] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:01:32.013113 25825 raft_consensus.cc:348] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f8c5e5b7eb41189fb255d61ef1a12c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43961 } } peers { permanent_uuid: "6d83eb2d74b54f778234af973d723126" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37745 } } peers { permanent_uuid: "aec0ee4a13094c26b3c2e4dd6ef93a53" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40021 } }
I20240501 20:01:32.013275 25825 raft_consensus.cc:374] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:32.013317 25825 raft_consensus.cc:729] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f8c5e5b7eb41189fb255d61ef1a12c, State: Initialized, Role: FOLLOWER
I20240501 20:01:32.013676 25825 consensus_queue.cc:260] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f8c5e5b7eb41189fb255d61ef1a12c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43961 } } peers { permanent_uuid: "6d83eb2d74b54f778234af973d723126" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37745 } } peers { permanent_uuid: "aec0ee4a13094c26b3c2e4dd6ef93a53" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40021 } }
I20240501 20:01:32.014225 25670 heartbeater.cc:503] Master 127.8.168.126:35575 was elected leader, sending a full tablet report...
I20240501 20:01:32.014307 25825 ts_tablet_manager.cc:1398] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20240501 20:01:32.051852 25537 tablet.cc:2369] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:01:32.202235 25805 tablet.cc:2369] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:01:32.207914 25827 raft_consensus.cc:482] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:32.208108 25827 raft_consensus.cc:504] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f8c5e5b7eb41189fb255d61ef1a12c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43961 } } peers { permanent_uuid: "6d83eb2d74b54f778234af973d723126" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37745 } } peers { permanent_uuid: "aec0ee4a13094c26b3c2e4dd6ef93a53" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40021 } }
I20240501 20:01:32.209245 25827 leader_election.cc:288] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42f8c5e5b7eb41189fb255d61ef1a12c (127.8.168.66:43961), aec0ee4a13094c26b3c2e4dd6ef93a53 (127.8.168.67:40021)
I20240501 20:01:32.212169 25830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:32.209534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:40021 (local address 127.8.168.65:48091)
0501 20:01:32.209846 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:32.209857 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:32.209965 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:32.210202 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:32.210205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:32.210252 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:32.210485 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:32.210492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.211211 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:32.211214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:32.211794 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:32.211803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.211895 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:32.211913 (+ 18us) client_negotiation.cc:769] Sending connection context
0501 20:01:32.211953 (+ 40us) client_negotiation.cc:240] Negotiation successful
0501 20:01:32.212011 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":88,"threads_started":1}
I20240501 20:01:32.212318 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:32.209505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:43961 (local address 127.8.168.65:56331)
0501 20:01:32.209826 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:32.209837 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:32.209940 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:32.210201 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:32.210204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:32.210220 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:32.210488 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:32.210495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.211272 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:32.211275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:32.211842 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:32.211860 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.211984 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:32.212002 (+ 18us) client_negotiation.cc:769] Sending connection context
0501 20:01:32.212048 (+ 46us) client_negotiation.cc:240] Negotiation successful
0501 20:01:32.212104 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"spinlock_wait_cycles":1920,"thread_start_us":102,"threads_started":1}
I20240501 20:01:32.212656 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:32.209621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:48091 (local address 127.8.168.67:40021)
0501 20:01:32.209740 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:32.209744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:32.209945 (+ 201us) server_negotiation.cc:407] Connection header received
0501 20:01:32.210067 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:32.210070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:32.210110 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:32.210175 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:32.210601 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.211098 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:32.211927 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.212379 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:32.212434 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:32.212486 (+ 52us) server_negotiation.cc:299] Negotiation successful
0501 20:01:32.212547 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20240501 20:01:32.212739 25818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:32.209599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:56331 (local address 127.8.168.66:43961)
0501 20:01:32.209731 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:32.209735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:32.209923 (+ 188us) server_negotiation.cc:407] Connection header received
0501 20:01:32.210059 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:32.210062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:32.210107 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:32.210169 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:32.210606 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.211131 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:32.211994 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.212451 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:32.212488 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:32.212559 (+ 71us) server_negotiation.cc:299] Negotiation successful
0501 20:01:32.212614 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240501 20:01:32.213241 25758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d718822eadd047d4916e00d3ac94a99f" candidate_uuid: "6d83eb2d74b54f778234af973d723126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec0ee4a13094c26b3c2e4dd6ef93a53" is_pre_election: true
I20240501 20:01:32.213258 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d718822eadd047d4916e00d3ac94a99f" candidate_uuid: "6d83eb2d74b54f778234af973d723126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f8c5e5b7eb41189fb255d61ef1a12c" is_pre_election: true
I20240501 20:01:32.213497 25758 raft_consensus.cc:2454] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d83eb2d74b54f778234af973d723126 in term 0.
I20240501 20:01:32.213506 25624 raft_consensus.cc:2454] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d83eb2d74b54f778234af973d723126 in term 0.
I20240501 20:01:32.213924 25423 leader_election.cc:302] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42f8c5e5b7eb41189fb255d61ef1a12c, 6d83eb2d74b54f778234af973d723126; no voters:
I20240501 20:01:32.214144 25827 raft_consensus.cc:2781] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:32.214215 25827 raft_consensus.cc:482] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:32.214251 25827 raft_consensus.cc:3037] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:32.215163 25827 raft_consensus.cc:504] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f8c5e5b7eb41189fb255d61ef1a12c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43961 } } peers { permanent_uuid: "6d83eb2d74b54f778234af973d723126" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37745 } } peers { permanent_uuid: "aec0ee4a13094c26b3c2e4dd6ef93a53" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40021 } }
I20240501 20:01:32.215528 25827 leader_election.cc:288] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [CANDIDATE]: Term 1 election: Requested vote from peers 42f8c5e5b7eb41189fb255d61ef1a12c (127.8.168.66:43961), aec0ee4a13094c26b3c2e4dd6ef93a53 (127.8.168.67:40021)
I20240501 20:01:32.215844 25624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d718822eadd047d4916e00d3ac94a99f" candidate_uuid: "6d83eb2d74b54f778234af973d723126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42f8c5e5b7eb41189fb255d61ef1a12c"
I20240501 20:01:32.215911 25758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d718822eadd047d4916e00d3ac94a99f" candidate_uuid: "6d83eb2d74b54f778234af973d723126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec0ee4a13094c26b3c2e4dd6ef93a53"
I20240501 20:01:32.215947 25624 raft_consensus.cc:3037] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:32.216035 25758 raft_consensus.cc:3037] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:32.216915 25624 raft_consensus.cc:2454] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d83eb2d74b54f778234af973d723126 in term 1.
I20240501 20:01:32.217113 25758 raft_consensus.cc:2454] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d83eb2d74b54f778234af973d723126 in term 1.
I20240501 20:01:32.217221 25423 leader_election.cc:302] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42f8c5e5b7eb41189fb255d61ef1a12c, 6d83eb2d74b54f778234af973d723126; no voters:
I20240501 20:01:32.217527 25827 raft_consensus.cc:2781] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:32.217749 25827 raft_consensus.cc:686] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [term 1 LEADER]: Becoming Leader. State: Replica: 6d83eb2d74b54f778234af973d723126, State: Running, Role: LEADER
I20240501 20:01:32.218093 25827 consensus_queue.cc:237] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f8c5e5b7eb41189fb255d61ef1a12c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43961 } } peers { permanent_uuid: "6d83eb2d74b54f778234af973d723126" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37745 } } peers { permanent_uuid: "aec0ee4a13094c26b3c2e4dd6ef93a53" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40021 } }
I20240501 20:01:32.221189 25345 catalog_manager.cc:5484] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d83eb2d74b54f778234af973d723126 (127.8.168.65). New cstate: current_term: 1 leader_uuid: "6d83eb2d74b54f778234af973d723126" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42f8c5e5b7eb41189fb255d61ef1a12c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d83eb2d74b54f778234af973d723126" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 37745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aec0ee4a13094c26b3c2e4dd6ef93a53" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40021 } health_report { overall_health: UNKNOWN } } }
W20240501 20:01:32.255955 25671 tablet.cc:2369] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:01:32.283775 25820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:32.280520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.8.168.65:37745)
0501 20:01:32.280693 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:32.280697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:32.280711 (+ 14us) server_negotiation.cc:407] Connection header received
0501 20:01:32.280855 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:32.280857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:32.280915 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:32.281013 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:01:32.281327 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.281876 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:32.282515 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.282682 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:32.282783 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:01:32.283294 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:01:32.283390 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:32.283544 (+ 154us) server_negotiation.cc:299] Negotiation successful
0501 20:01:32.283599 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20240501 20:01:32.289440 25758 raft_consensus.cc:1261] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53 [term 1 FOLLOWER]: Refusing update from remote peer 6d83eb2d74b54f778234af973d723126: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:32.289443 25624 raft_consensus.cc:1261] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c [term 1 FOLLOWER]: Refusing update from remote peer 6d83eb2d74b54f778234af973d723126: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:32.290083 25827 consensus_queue.cc:1035] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec0ee4a13094c26b3c2e4dd6ef93a53" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:32.290266 25832 consensus_queue.cc:1035] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42f8c5e5b7eb41189fb255d61ef1a12c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 43961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:32.298393 25837 mvcc.cc:205] Tried to move back new op lower bound from 7022975763610251264 to 7022975763328692224. Current Snapshot: MvccSnapshot[applied={T|T < 7022975763610251264}]
I20240501 20:01:32.306269 25344 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47808:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:01:32.306463 25344 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47808:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:01:32.308825 25344 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c20efebf24614e26a3fcc3e6cc4e49d3: Sending DeleteTablet for 3 replicas of tablet d718822eadd047d4916e00d3ac94a99f
I20240501 20:01:32.309509 25470 tablet_service.cc:1514] Processing DeleteTablet for tablet d718822eadd047d4916e00d3ac94a99f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:32 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:36658
I20240501 20:01:32.309507 25603 tablet_service.cc:1514] Processing DeleteTablet for tablet d718822eadd047d4916e00d3ac94a99f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:32 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:44820
I20240501 20:01:32.309638 25738 tablet_service.cc:1514] Processing DeleteTablet for tablet d718822eadd047d4916e00d3ac94a99f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:32 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:38772
I20240501 20:01:32.310195 25845 tablet_replica.cc:330] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126: stopping tablet replica
I20240501 20:01:32.310194 25846 tablet_replica.cc:330] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c: stopping tablet replica
I20240501 20:01:32.310459 25845 raft_consensus.cc:2229] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:32.310458 25846 raft_consensus.cc:2229] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:32.310683 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 25407
I20240501 20:01:32.310822 25845 raft_consensus.cc:2258] T d718822eadd047d4916e00d3ac94a99f P 6d83eb2d74b54f778234af973d723126 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:32.311331 25846 raft_consensus.cc:2258] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:32.311722 25847 tablet_replica.cc:330] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53: stopping tablet replica
I20240501 20:01:32.312002 25847 raft_consensus.cc:2229] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:32.312295 25847 raft_consensus.cc:2258] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:32.313459 25846 ts_tablet_manager.cc:1875] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:01:32.313522 25847 ts_tablet_manager.cc:1875] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:01:32.315460 25846 ts_tablet_manager.cc:1888] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:01:32.315541 25846 log.cc:1196] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/d718822eadd047d4916e00d3ac94a99f
I20240501 20:01:32.315804 25846 ts_tablet_manager.cc:1909] T d718822eadd047d4916e00d3ac94a99f P 42f8c5e5b7eb41189fb255d61ef1a12c: Deleting consensus metadata
I20240501 20:01:32.315809 25847 ts_tablet_manager.cc:1888] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:01:32.315888 25847 log.cc:1196] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/d718822eadd047d4916e00d3ac94a99f
I20240501 20:01:32.316179 25847 ts_tablet_manager.cc:1909] T d718822eadd047d4916e00d3ac94a99f P aec0ee4a13094c26b3c2e4dd6ef93a53: Deleting consensus metadata
I20240501 20:01:32.316938 25329 catalog_manager.cc:4830] TS 42f8c5e5b7eb41189fb255d61ef1a12c (127.8.168.66:43961): tablet d718822eadd047d4916e00d3ac94a99f (table test-table [id=ac7e3ebb44e349129686fe2d938aab71]) successfully deleted
I20240501 20:01:32.317219 25332 catalog_manager.cc:4830] TS aec0ee4a13094c26b3c2e4dd6ef93a53 (127.8.168.67:40021): tablet d718822eadd047d4916e00d3ac94a99f (table test-table [id=ac7e3ebb44e349129686fe2d938aab71]) successfully deleted
W20240501 20:01:32.317762 25332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.168.65:37745 (error 108)
I20240501 20:01:32.317881 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 25541
I20240501 20:01:32.319099 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:32.318813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:37745 (local address 127.0.0.1:36682)
0501 20:01:32.318953 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:32.319016 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:37745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20240501 20:01:32.319237 25332 catalog_manager.cc:4573] TS 6d83eb2d74b54f778234af973d723126 (127.8.168.65:37745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d718822eadd047d4916e00d3ac94a99f: Network error: Client connection negotiation failed: client connection to 127.8.168.65:37745: connect: Connection refused (error 111)
I20240501 20:01:32.323385 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 25675
I20240501 20:01:32.327905 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 25313
2024-05-01T20:01:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2246 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2024-05-01T20:01:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:32Z Disabled control of system clock
I20240501 20:01:32.354131 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:42397
--webserver_interface=127.8.168.126
--webserver_port=0
--builtin_ntp_servers=127.8.168.84:35025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:42397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240501 20:01:32.456981 25852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:32.457192 25852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:32.457237 25852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:32.460232 25852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:32.460291 25852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:32.460306 25852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:32.460322 25852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:32.460338 25852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:32.463871 25852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:42397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:42397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:32.464709 25852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:32.465822 25852 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:32.472371 25857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:32.472702 25860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:32.473251 25858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:32.473639 25852 server_base.cc:1034] running on GCE node
I20240501 20:01:32.474056 25852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:32.474984 25852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:32.476156 25852 hybrid_clock.cc:648] HybridClock initialized: now 1714593692476142 us; error 24 us; skew 500 ppm
I20240501 20:01:32.477970 25852 webserver.cc:460] Webserver started at http://127.8.168.126:39843/ using document root <none> and password file <none>
I20240501 20:01:32.478582 25852 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:32.478636 25852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:32.478823 25852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:32.480504 25852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "914f7701b4714d15a74c3e8ea08b6bef"
format_stamp: "Formatted at 2024-05-01 20:01:32 on dist-test-slave-tzct"
server_key: "102b6bdc0970eeb2cac908288f1e5607"
server_key_iv: "ee29211ac0ec000959529928db447a37"
server_key_version: "encryptionkey@0"
I20240501 20:01:32.480976 25852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "914f7701b4714d15a74c3e8ea08b6bef"
format_stamp: "Formatted at 2024-05-01 20:01:32 on dist-test-slave-tzct"
server_key: "102b6bdc0970eeb2cac908288f1e5607"
server_key_iv: "ee29211ac0ec000959529928db447a37"
server_key_version: "encryptionkey@0"
I20240501 20:01:32.484038 25852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:01:32.486285 25866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:32.487285 25852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:01:32.487396 25852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "914f7701b4714d15a74c3e8ea08b6bef"
format_stamp: "Formatted at 2024-05-01 20:01:32 on dist-test-slave-tzct"
server_key: "102b6bdc0970eeb2cac908288f1e5607"
server_key_iv: "ee29211ac0ec000959529928db447a37"
server_key_version: "encryptionkey@0"
I20240501 20:01:32.487483 25852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:32.514801 25852 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:32.515496 25852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:32.515666 25852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:32.521929 25852 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:42397
I20240501 20:01:32.521942 25918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:42397 every 8 connection(s)
I20240501 20:01:32.522861 25852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:32.525297 25919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:32.529029 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 25852
I20240501 20:01:32.529155 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:32.529431 8865 external_mini_cluster.cc:1428] Setting key 3a0141f6235ac498e0e32202a5347c2d
I20240501 20:01:32.530949 25919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef: Bootstrap starting.
I20240501 20:01:32.533062 25919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:32.533726 25919 log.cc:824] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:32.535471 25919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef: No bootstrap required, opened a new log
I20240501 20:01:32.537240 25922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:32.530741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58062 (local address 127.8.168.126:42397)
0501 20:01:32.531394 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:32.531402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:32.531442 (+ 40us) server_negotiation.cc:407] Connection header received
0501 20:01:32.532015 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:32.532041 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:32.532386 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:32.532675 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:32.533263 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.534153 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:32.534882 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.535140 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:32.535510 (+ 370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:32.535531 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:32.535543 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:32.535985 (+ 442us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:32.536036 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:32.536055 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:32.536163 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:32.536295 (+ 132us) server_negotiation.cc:299] Negotiation successful
0501 20:01:32.536547 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":123,"threads_started":1}
I20240501 20:01:32.538897 25919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914f7701b4714d15a74c3e8ea08b6bef" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42397 } }
I20240501 20:01:32.539101 25919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:32.539161 25919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914f7701b4714d15a74c3e8ea08b6bef, State: Initialized, Role: FOLLOWER
I20240501 20:01:32.539530 25919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914f7701b4714d15a74c3e8ea08b6bef" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42397 } }
I20240501 20:01:32.539649 25919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:32.539692 25919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:32.539767 25919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:32.540484 25919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914f7701b4714d15a74c3e8ea08b6bef" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42397 } }
I20240501 20:01:32.540787 25919 leader_election.cc:302] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 914f7701b4714d15a74c3e8ea08b6bef; no voters:
I20240501 20:01:32.541044 25919 leader_election.cc:288] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:32.541137 25924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:32.541333 25924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [term 1 LEADER]: Becoming Leader. State: Replica: 914f7701b4714d15a74c3e8ea08b6bef, State: Running, Role: LEADER
I20240501 20:01:32.541607 25924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914f7701b4714d15a74c3e8ea08b6bef" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42397 } }
I20240501 20:01:32.542057 25919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:32.544759 25925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "914f7701b4714d15a74c3e8ea08b6bef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914f7701b4714d15a74c3e8ea08b6bef" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42397 } } }
I20240501 20:01:32.545017 25925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:32.544900 25926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 914f7701b4714d15a74c3e8ea08b6bef. Latest consensus state: current_term: 1 leader_uuid: "914f7701b4714d15a74c3e8ea08b6bef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914f7701b4714d15a74c3e8ea08b6bef" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42397 } } }
I20240501 20:01:32.545228 25926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef [sys.catalog]: This master's current role is: LEADER
W20240501 20:01:32.547231 25939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:01:32.547300 25939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:01:32.547360 25940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:32.548050 25940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:32.552868 25940 catalog_manager.cc:1347] Generated new cluster ID: 247d35caab834fd6874153ea2b7bb635
I20240501 20:01:32.552964 25940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:32.559228 25940 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:32.560375 25940 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:32.574402 25940 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef: Generated new TSK 0
I20240501 20:01:32.575232 25940 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:32.581599 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:42397
--builtin_ntp_servers=127.8.168.84:35025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240501 20:01:32.683151 25943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:32.683367 25943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:32.683437 25943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:32.686450 25943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:32.686501 25943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:32.686589 25943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 20:01:32.690099 25943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:42397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:32.690949 25943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:32.692044 25943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:32.699496 25948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:32.699927 25943 server_base.cc:1034] running on GCE node
W20240501 20:01:32.700316 25951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:32.700621 25949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:32.701030 25943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:32.701521 25943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:32.702687 25943 hybrid_clock.cc:648] HybridClock initialized: now 1714593692702674 us; error 24 us; skew 500 ppm
I20240501 20:01:32.704443 25943 webserver.cc:460] Webserver started at http://127.8.168.65:41827/ using document root <none> and password file <none>
I20240501 20:01:32.704998 25943 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:32.705052 25943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:32.705221 25943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:32.706893 25943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "e237431e49a34f5ab47d3f33eeef2243"
format_stamp: "Formatted at 2024-05-01 20:01:32 on dist-test-slave-tzct"
server_key: "3bdc52c2b20443030413d705a2152fca"
server_key_iv: "f940866728513a55ddb37fa7847d7eda"
server_key_version: "encryptionkey@0"
I20240501 20:01:32.707319 25943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "e237431e49a34f5ab47d3f33eeef2243"
format_stamp: "Formatted at 2024-05-01 20:01:32 on dist-test-slave-tzct"
server_key: "3bdc52c2b20443030413d705a2152fca"
server_key_iv: "f940866728513a55ddb37fa7847d7eda"
server_key_version: "encryptionkey@0"
I20240501 20:01:32.710383 25943 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240501 20:01:32.712371 25957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:32.713368 25943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:32.713465 25943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "e237431e49a34f5ab47d3f33eeef2243"
format_stamp: "Formatted at 2024-05-01 20:01:32 on dist-test-slave-tzct"
server_key: "3bdc52c2b20443030413d705a2152fca"
server_key_iv: "f940866728513a55ddb37fa7847d7eda"
server_key_version: "encryptionkey@0"
I20240501 20:01:32.713544 25943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:32.734552 25943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:32.735175 25943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:32.735329 25943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:32.735852 25943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:32.736707 25943 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:32.736776 25943 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:32.736846 25943 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:32.736877 25943 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:32.745962 25943 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:45351
I20240501 20:01:32.746004 26070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:45351 every 8 connection(s)
I20240501 20:01:32.746848 25943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 20:01:32.754263 25922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:32.748489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:50135 (local address 127.8.168.126:42397)
0501 20:01:32.748629 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:32.748632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:32.749500 (+ 868us) server_negotiation.cc:407] Connection header received
0501 20:01:32.750262 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:32.750265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:32.750321 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:32.750407 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:32.751699 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.752202 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:32.752833 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.753015 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:32.753772 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:32.753792 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:32.753794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:32.753806 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:32.753825 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:32.753832 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:32.753891 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:32.754044 (+ 153us) server_negotiation.cc:299] Negotiation successful
0501 20:01:32.754116 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20240501 20:01:32.754873 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:32.748772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:42397 (local address 127.8.168.65:50135)
0501 20:01:32.749369 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:32.749400 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:32.750067 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:32.750558 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:32.750566 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:32.750638 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:32.751554 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:32.751568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.752323 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:32.752326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:32.752701 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:32.752708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.752952 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:32.753549 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 20:01:32.753551 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:32.753584 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 20:01:32.753639 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:32.753910 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:32.753918 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:32.753923 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 20:01:32.754033 (+ 110us) client_negotiation.cc:240] Negotiation successful
0501 20:01:32.754255 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":96,"threads_started":1}
I20240501 20:01:32.756096 26071 heartbeater.cc:347] Connected to a master server at 127.8.168.126:42397
I20240501 20:01:32.756296 26071 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:32.756790 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 25943
I20240501 20:01:32.756831 26071 heartbeater.cc:511] Master 127.8.168.126:42397 requested a full tablet report, sending...
I20240501 20:01:32.756911 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 20:01:32.757174 8865 external_mini_cluster.cc:1428] Setting key 11f678e8982e69292e39fd2f883f05e0
I20240501 20:01:32.758317 25883 ts_manager.cc:194] Registered new tserver with Master: e237431e49a34f5ab47d3f33eeef2243 (127.8.168.65:45351)
I20240501 20:01:32.759320 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:42397
--builtin_ntp_servers=127.8.168.84:35025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240501 20:01:32.759807 25883 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.168.65:50135
W20240501 20:01:32.861740 26074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:32.861949 26074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:32.861994 26074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:32.864966 26074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:32.865010 26074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:32.865077 26074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 20:01:32.868513 26074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:42397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:32.869300 26074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:32.870347 26074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:32.878211 26082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:32.878330 26080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:32.878711 26079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:32.879235 26074 server_base.cc:1034] running on GCE node
I20240501 20:01:32.879573 26074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:32.880084 26074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:32.881227 26074 hybrid_clock.cc:648] HybridClock initialized: now 1714593692881204 us; error 35 us; skew 500 ppm
I20240501 20:01:32.882966 26074 webserver.cc:460] Webserver started at http://127.8.168.66:43653/ using document root <none> and password file <none>
I20240501 20:01:32.883456 26074 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:32.883503 26074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:32.883653 26074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:32.885383 26074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "bd3e3175ddc94b50b362777b9e26511b"
format_stamp: "Formatted at 2024-05-01 20:01:32 on dist-test-slave-tzct"
server_key: "27a7d74399d6f09c4354f429c3c3c68b"
server_key_iv: "f1f99a5bc3d3843389cc533eccb3ba9f"
server_key_version: "encryptionkey@0"
I20240501 20:01:32.885795 26074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "bd3e3175ddc94b50b362777b9e26511b"
format_stamp: "Formatted at 2024-05-01 20:01:32 on dist-test-slave-tzct"
server_key: "27a7d74399d6f09c4354f429c3c3c68b"
server_key_iv: "f1f99a5bc3d3843389cc533eccb3ba9f"
server_key_version: "encryptionkey@0"
I20240501 20:01:32.888779 26074 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:01:32.890920 26088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:32.891933 26074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:32.892035 26074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "bd3e3175ddc94b50b362777b9e26511b"
format_stamp: "Formatted at 2024-05-01 20:01:32 on dist-test-slave-tzct"
server_key: "27a7d74399d6f09c4354f429c3c3c68b"
server_key_iv: "f1f99a5bc3d3843389cc533eccb3ba9f"
server_key_version: "encryptionkey@0"
I20240501 20:01:32.892127 26074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:32.907585 26074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:32.908211 26074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:32.908347 26074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:32.908890 26074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:32.909745 26074 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:32.909790 26074 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:32.909850 26074 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:32.909879 26074 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:32.919263 26074 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:35615
I20240501 20:01:32.919292 26201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:35615 every 8 connection(s)
I20240501 20:01:32.920167 26074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 20:01:32.924619 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 26074
I20240501 20:01:32.924791 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 20:01:32.925086 8865 external_mini_cluster.cc:1428] Setting key 0d8dfd69b3fcdab6697ede03e9e9eca1
I20240501 20:01:32.927074 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:42397
--builtin_ntp_servers=127.8.168.84:35025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240501 20:01:32.927919 25922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:32.921777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:33669 (local address 127.8.168.126:42397)
0501 20:01:32.921967 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:32.921970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:32.922782 (+ 812us) server_negotiation.cc:407] Connection header received
0501 20:01:32.923551 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:32.923554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:32.923604 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:32.923684 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:32.925200 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.925782 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:32.926541 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.926673 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:32.927501 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:32.927519 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:32.927521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:32.927533 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:32.927553 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:32.927560 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:32.927618 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:32.927761 (+ 143us) server_negotiation.cc:299] Negotiation successful
0501 20:01:32.927810 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20240501 20:01:32.928704 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:32.922064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:42397 (local address 127.8.168.66:33669)
0501 20:01:32.922649 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:32.922682 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:32.923328 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:32.923826 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:32.923833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:32.923908 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:32.925029 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:32.925049 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.925933 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:32.925937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:32.926420 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:32.926429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:32.926634 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:32.927242 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 20:01:32.927244 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:32.927290 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 20:01:32.927353 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:32.927634 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:32.927641 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:32.927646 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 20:01:32.927768 (+ 122us) client_negotiation.cc:240] Negotiation successful
0501 20:01:32.928008 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":95,"threads_started":1}
I20240501 20:01:32.929698 26202 heartbeater.cc:347] Connected to a master server at 127.8.168.126:42397
I20240501 20:01:32.929911 26202 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:32.930402 26202 heartbeater.cc:511] Master 127.8.168.126:42397 requested a full tablet report, sending...
I20240501 20:01:32.931339 25883 ts_manager.cc:194] Registered new tserver with Master: bd3e3175ddc94b50b362777b9e26511b (127.8.168.66:35615)
I20240501 20:01:32.932104 25883 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.168.66:33669
W20240501 20:01:33.029259 26205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:33.029458 26205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:33.029503 26205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:33.032436 26205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:33.032480 26205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:33.032544 26205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 20:01:33.035979 26205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:42397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:33.036758 26205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:33.037803 26205 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:33.045195 26210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:33.045591 26211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:33.045969 26213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:33.046494 26205 server_base.cc:1034] running on GCE node
I20240501 20:01:33.046888 26205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:33.047410 26205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:33.048547 26205 hybrid_clock.cc:648] HybridClock initialized: now 1714593693048533 us; error 29 us; skew 500 ppm
I20240501 20:01:33.050192 26205 webserver.cc:460] Webserver started at http://127.8.168.67:34907/ using document root <none> and password file <none>
I20240501 20:01:33.050712 26205 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:33.050768 26205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:33.050923 26205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:33.052599 26205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "f9884f23a90142ebb9db9bc679df0020"
format_stamp: "Formatted at 2024-05-01 20:01:33 on dist-test-slave-tzct"
server_key: "5c0e349e655bf69689f7ed6327d9e514"
server_key_iv: "9be7653173f2a4825ebf90dbd83d13aa"
server_key_version: "encryptionkey@0"
I20240501 20:01:33.053084 26205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "f9884f23a90142ebb9db9bc679df0020"
format_stamp: "Formatted at 2024-05-01 20:01:33 on dist-test-slave-tzct"
server_key: "5c0e349e655bf69689f7ed6327d9e514"
server_key_iv: "9be7653173f2a4825ebf90dbd83d13aa"
server_key_version: "encryptionkey@0"
I20240501 20:01:33.056142 26205 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:01:33.058279 26219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:33.059192 26205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:01:33.059290 26205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "f9884f23a90142ebb9db9bc679df0020"
format_stamp: "Formatted at 2024-05-01 20:01:33 on dist-test-slave-tzct"
server_key: "5c0e349e655bf69689f7ed6327d9e514"
server_key_iv: "9be7653173f2a4825ebf90dbd83d13aa"
server_key_version: "encryptionkey@0"
I20240501 20:01:33.059373 26205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:33.076598 26205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:33.077214 26205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:33.077354 26205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:33.077881 26205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:33.078739 26205 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:33.078785 26205 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:33.078836 26205 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:33.078861 26205 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:33.086998 26205 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:39639
I20240501 20:01:33.087078 26332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:39639 every 8 connection(s)
I20240501 20:01:33.087881 26205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 20:01:33.092509 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 26205
I20240501 20:01:33.092624 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 20:01:33.092955 8865 external_mini_cluster.cc:1428] Setting key 76241eb44f71dcbca3ddc7490df3cf3e
I20240501 20:01:33.095548 25922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:33.089476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:48865 (local address 127.8.168.126:42397)
0501 20:01:33.089601 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:33.089604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:33.090488 (+ 884us) server_negotiation.cc:407] Connection header received
0501 20:01:33.091271 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:33.091274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:33.091331 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:33.091411 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:33.092754 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.093418 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:33.094146 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.094337 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:33.095085 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:33.095103 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:33.095105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:33.095116 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:33.095135 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:33.095142 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:33.095200 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:33.095340 (+ 140us) server_negotiation.cc:299] Negotiation successful
0501 20:01:33.095408 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20240501 20:01:33.096143 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:33.089731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:42397 (local address 127.8.168.67:48865)
0501 20:01:33.090360 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:33.090394 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:33.091084 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:33.091561 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:33.091568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:33.091642 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:33.092572 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:33.092586 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.093537 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:33.093541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:33.094027 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:33.094037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.094219 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:33.094823 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 20:01:33.094825 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:33.094869 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 20:01:33.094940 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:33.095210 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:33.095217 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:33.095223 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 20:01:33.095339 (+ 116us) client_negotiation.cc:240] Negotiation successful
0501 20:01:33.095550 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":132,"threads_started":1}
I20240501 20:01:33.097043 26333 heartbeater.cc:347] Connected to a master server at 127.8.168.126:42397
I20240501 20:01:33.097245 26333 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:33.097729 26333 heartbeater.cc:511] Master 127.8.168.126:42397 requested a full tablet report, sending...
I20240501 20:01:33.098603 25883 ts_manager.cc:194] Registered new tserver with Master: f9884f23a90142ebb9db9bc679df0020 (127.8.168.67:39639)
I20240501 20:01:33.099128 25883 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.168.67:48865
I20240501 20:01:33.106403 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 20:01:33.111199 25922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:33.108054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58066 (local address 127.8.168.126:42397)
0501 20:01:33.108186 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:33.108190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:33.108337 (+ 147us) server_negotiation.cc:407] Connection header received
0501 20:01:33.108443 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:33.108446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:33.108490 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:33.108553 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:33.108970 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.109447 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:33.110040 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.110214 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:33.110724 (+ 510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:33.110739 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:33.110741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:33.110752 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:33.110770 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:33.110777 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:33.110836 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:33.111015 (+ 179us) server_negotiation.cc:299] Negotiation successful
0501 20:01:33.111061 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240501 20:01:33.113579 25883 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:01:33.116019 25883 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:01:33.128337 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:33.124009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:35615 (local address 127.0.0.1:33654)
0501 20:01:33.124656 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:33.124669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:33.124759 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:33.125805 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:33.125808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:33.125823 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:33.126021 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:33.126026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.127064 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:33.127067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:33.127856 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:33.127865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.127974 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:33.128027 (+ 53us) client_negotiation.cc:769] Sending connection context
0501 20:01:33.128110 (+ 83us) client_negotiation.cc:240] Negotiation successful
0501 20:01:33.128190 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":391,"thread_start_us":77,"threads_started":1}
I20240501 20:01:33.128337 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:33.124009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:45351 (local address 127.0.0.1:40672)
0501 20:01:33.124314 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:33.124353 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:33.124480 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:33.125445 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:33.125452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:33.125471 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:33.125770 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:33.125780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.127075 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:33.127078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:33.127902 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:33.127911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.127988 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:33.128032 (+ 44us) client_negotiation.cc:769] Sending connection context
0501 20:01:33.128105 (+ 73us) client_negotiation.cc:240] Negotiation successful
0501 20:01:33.128196 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":73,"threads_started":1}
I20240501 20:01:33.128768 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:33.124744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:39639 (local address 127.0.0.1:48462)
0501 20:01:33.125007 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:33.125018 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:33.125086 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:33.125738 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:33.125741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:33.125760 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:33.126019 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:33.126026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.127324 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:33.127328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:33.128125 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:33.128132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.128217 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:33.128230 (+ 13us) client_negotiation.cc:769] Sending connection context
0501 20:01:33.128272 (+ 42us) client_negotiation.cc:240] Negotiation successful
0501 20:01:33.128320 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":79,"threads_started":1}
I20240501 20:01:33.129393 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:33.124153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.8.168.66:35615)
0501 20:01:33.125384 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:33.125401 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:33.125423 (+ 22us) server_negotiation.cc:407] Connection header received
0501 20:01:33.125504 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:33.125508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:33.125664 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:33.126154 (+ 490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:33.126255 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.126959 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:33.128386 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.128946 (+ 560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:33.129108 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:33.129173 (+ 65us) server_negotiation.cc:299] Negotiation successful
0501 20:01:33.129259 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135,"thread_start_us":129,"threads_started":1}
I20240501 20:01:33.129757 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:33.124677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40672 (local address 127.8.168.65:45351)
0501 20:01:33.124983 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:33.124990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:33.125014 (+ 24us) server_negotiation.cc:407] Connection header received
0501 20:01:33.125109 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:33.125116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:33.125297 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:33.125455 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:33.125889 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.126955 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:33.128936 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.129368 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:33.129480 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:33.129550 (+ 70us) server_negotiation.cc:299] Negotiation successful
0501 20:01:33.129633 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":128,"threads_started":1}
I20240501 20:01:33.130488 26349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:33.125005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48462 (local address 127.8.168.67:39639)
0501 20:01:33.125331 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:33.125337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:33.125358 (+ 21us) server_negotiation.cc:407] Connection header received
0501 20:01:33.125439 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:33.125444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:33.125589 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:33.125733 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:33.126135 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.127230 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:33.129032 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.129550 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:33.129658 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:33.129711 (+ 53us) server_negotiation.cc:299] Negotiation successful
0501 20:01:33.129790 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":106,"threads_started":1}
I20240501 20:01:33.131897 26136 tablet_service.cc:1467] Processing CreateTablet for tablet e10f4d36790d4f628a5cc1c79db3657d (DEFAULT_TABLE table=test-table [id=465c81f6cf5b46a3adcc8b4f2870ac2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:33.132071 26005 tablet_service.cc:1467] Processing CreateTablet for tablet e10f4d36790d4f628a5cc1c79db3657d (DEFAULT_TABLE table=test-table [id=465c81f6cf5b46a3adcc8b4f2870ac2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:33.132740 26136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e10f4d36790d4f628a5cc1c79db3657d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:33.132566 26267 tablet_service.cc:1467] Processing CreateTablet for tablet e10f4d36790d4f628a5cc1c79db3657d (DEFAULT_TABLE table=test-table [id=465c81f6cf5b46a3adcc8b4f2870ac2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:33.132983 26005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e10f4d36790d4f628a5cc1c79db3657d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:33.133356 26267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e10f4d36790d4f628a5cc1c79db3657d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:33.138036 26350 tablet_bootstrap.cc:492] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b: Bootstrap starting.
I20240501 20:01:33.139520 26352 tablet_bootstrap.cc:492] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020: Bootstrap starting.
I20240501 20:01:33.139643 26351 tablet_bootstrap.cc:492] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243: Bootstrap starting.
I20240501 20:01:33.140832 26350 tablet_bootstrap.cc:654] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:33.141206 26352 tablet_bootstrap.cc:654] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:33.141553 26351 tablet_bootstrap.cc:654] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:33.141642 26350 log.cc:824] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:33.141827 26352 log.cc:824] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:33.142261 26351 log.cc:824] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:33.143570 26350 tablet_bootstrap.cc:492] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b: No bootstrap required, opened a new log
I20240501 20:01:33.143570 26352 tablet_bootstrap.cc:492] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020: No bootstrap required, opened a new log
I20240501 20:01:33.143714 26351 tablet_bootstrap.cc:492] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243: No bootstrap required, opened a new log
I20240501 20:01:33.143744 26350 ts_tablet_manager.cc:1367] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:01:33.143744 26352 ts_tablet_manager.cc:1367] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:01:33.143869 26351 ts_tablet_manager.cc:1367] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:01:33.146750 26351 raft_consensus.cc:348] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3e3175ddc94b50b362777b9e26511b" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35615 } } peers { permanent_uuid: "e237431e49a34f5ab47d3f33eeef2243" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45351 } } peers { permanent_uuid: "f9884f23a90142ebb9db9bc679df0020" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39639 } }
I20240501 20:01:33.146936 26351 raft_consensus.cc:374] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:33.147004 26351 raft_consensus.cc:729] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e237431e49a34f5ab47d3f33eeef2243, State: Initialized, Role: FOLLOWER
I20240501 20:01:33.147302 26350 raft_consensus.cc:348] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3e3175ddc94b50b362777b9e26511b" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35615 } } peers { permanent_uuid: "e237431e49a34f5ab47d3f33eeef2243" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45351 } } peers { permanent_uuid: "f9884f23a90142ebb9db9bc679df0020" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39639 } }
I20240501 20:01:33.147495 26350 raft_consensus.cc:374] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:33.147475 26351 consensus_queue.cc:260] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3e3175ddc94b50b362777b9e26511b" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35615 } } peers { permanent_uuid: "e237431e49a34f5ab47d3f33eeef2243" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45351 } } peers { permanent_uuid: "f9884f23a90142ebb9db9bc679df0020" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39639 } }
I20240501 20:01:33.147468 26352 raft_consensus.cc:348] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3e3175ddc94b50b362777b9e26511b" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35615 } } peers { permanent_uuid: "e237431e49a34f5ab47d3f33eeef2243" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45351 } } peers { permanent_uuid: "f9884f23a90142ebb9db9bc679df0020" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39639 } }
I20240501 20:01:33.147594 26350 raft_consensus.cc:729] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3e3175ddc94b50b362777b9e26511b, State: Initialized, Role: FOLLOWER
I20240501 20:01:33.147624 26352 raft_consensus.cc:374] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:33.147689 26352 raft_consensus.cc:729] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9884f23a90142ebb9db9bc679df0020, State: Initialized, Role: FOLLOWER
I20240501 20:01:33.148061 26352 consensus_queue.cc:260] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3e3175ddc94b50b362777b9e26511b" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35615 } } peers { permanent_uuid: "e237431e49a34f5ab47d3f33eeef2243" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45351 } } peers { permanent_uuid: "f9884f23a90142ebb9db9bc679df0020" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39639 } }
I20240501 20:01:33.148061 26350 consensus_queue.cc:260] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3e3175ddc94b50b362777b9e26511b" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35615 } } peers { permanent_uuid: "e237431e49a34f5ab47d3f33eeef2243" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45351 } } peers { permanent_uuid: "f9884f23a90142ebb9db9bc679df0020" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39639 } }
I20240501 20:01:33.148285 26351 ts_tablet_manager.cc:1398] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 20:01:33.148680 26202 heartbeater.cc:503] Master 127.8.168.126:42397 was elected leader, sending a full tablet report...
I20240501 20:01:33.148878 26350 ts_tablet_manager.cc:1398] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 20:01:33.148875 26352 ts_tablet_manager.cc:1398] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 20:01:33.148919 26333 heartbeater.cc:503] Master 127.8.168.126:42397 was elected leader, sending a full tablet report...
I20240501 20:01:33.149838 26071 heartbeater.cc:503] Master 127.8.168.126:42397 was elected leader, sending a full tablet report...
W20240501 20:01:33.171029 26203 tablet.cc:2369] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:01:33.247941 26072 tablet.cc:2369] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:01:33.338721 26334 tablet.cc:2369] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:01:33.366093 26356 raft_consensus.cc:482] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:33.366345 26356 raft_consensus.cc:504] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3e3175ddc94b50b362777b9e26511b" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35615 } } peers { permanent_uuid: "e237431e49a34f5ab47d3f33eeef2243" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45351 } } peers { permanent_uuid: "f9884f23a90142ebb9db9bc679df0020" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39639 } }
I20240501 20:01:33.367427 26356 leader_election.cc:288] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3e3175ddc94b50b362777b9e26511b (127.8.168.66:35615), f9884f23a90142ebb9db9bc679df0020 (127.8.168.67:39639)
I20240501 20:01:33.370618 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:33.367661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:35615 (local address 127.8.168.65:46181)
0501 20:01:33.367898 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:33.367932 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:33.368043 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:33.368322 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:33.368325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:33.368346 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:33.368632 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:33.368642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.369592 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:33.369596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:33.370296 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:33.370305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.370394 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:33.370411 (+ 17us) client_negotiation.cc:769] Sending connection context
0501 20:01:33.370446 (+ 35us) client_negotiation.cc:240] Negotiation successful
0501 20:01:33.370489 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":71,"threads_started":1}
I20240501 20:01:33.371161 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:33.367764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:46181 (local address 127.8.168.66:35615)
0501 20:01:33.367927 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:33.367931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:33.368008 (+ 77us) server_negotiation.cc:407] Connection header received
0501 20:01:33.368149 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:33.368153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:33.368214 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:33.368301 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:33.368780 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.369482 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:33.370434 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.370901 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:33.370939 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:33.370997 (+ 58us) server_negotiation.cc:299] Negotiation successful
0501 20:01:33.371049 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20240501 20:01:33.371181 26360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:33.368025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:39639 (local address 127.8.168.65:39039)
0501 20:01:33.368351 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:33.368366 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:33.368454 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:33.368757 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:33.368761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:33.368779 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:33.369101 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:33.369109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.369974 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:33.369977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:33.370694 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:33.370704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.370915 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:33.370931 (+ 16us) client_negotiation.cc:769] Sending connection context
0501 20:01:33.370981 (+ 50us) client_negotiation.cc:240] Negotiation successful
0501 20:01:33.371037 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":53,"threads_started":1}
I20240501 20:01:33.371498 26349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:33.368105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:39039 (local address 127.8.168.67:39639)
0501 20:01:33.368234 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:33.368237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:33.368453 (+ 216us) server_negotiation.cc:407] Connection header received
0501 20:01:33.368536 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:33.368539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:33.368585 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:33.368661 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:33.369217 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.369825 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:33.370909 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.371229 (+ 320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:33.371264 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:33.371319 (+ 55us) server_negotiation.cc:299] Negotiation successful
0501 20:01:33.371370 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 20:01:33.371784 26156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10f4d36790d4f628a5cc1c79db3657d" candidate_uuid: "e237431e49a34f5ab47d3f33eeef2243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3e3175ddc94b50b362777b9e26511b" is_pre_election: true
I20240501 20:01:33.372016 26287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10f4d36790d4f628a5cc1c79db3657d" candidate_uuid: "e237431e49a34f5ab47d3f33eeef2243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9884f23a90142ebb9db9bc679df0020" is_pre_election: true
I20240501 20:01:33.372103 26156 raft_consensus.cc:2454] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e237431e49a34f5ab47d3f33eeef2243 in term 0.
I20240501 20:01:33.372267 26287 raft_consensus.cc:2454] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e237431e49a34f5ab47d3f33eeef2243 in term 0.
I20240501 20:01:33.372566 25960 leader_election.cc:302] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3e3175ddc94b50b362777b9e26511b, e237431e49a34f5ab47d3f33eeef2243; no voters:
I20240501 20:01:33.372831 26356 raft_consensus.cc:2781] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:33.372902 26356 raft_consensus.cc:482] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:33.372931 26356 raft_consensus.cc:3037] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:33.373831 26356 raft_consensus.cc:504] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3e3175ddc94b50b362777b9e26511b" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35615 } } peers { permanent_uuid: "e237431e49a34f5ab47d3f33eeef2243" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45351 } } peers { permanent_uuid: "f9884f23a90142ebb9db9bc679df0020" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39639 } }
I20240501 20:01:33.374153 26356 leader_election.cc:288] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [CANDIDATE]: Term 1 election: Requested vote from peers bd3e3175ddc94b50b362777b9e26511b (127.8.168.66:35615), f9884f23a90142ebb9db9bc679df0020 (127.8.168.67:39639)
I20240501 20:01:33.374500 26156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10f4d36790d4f628a5cc1c79db3657d" candidate_uuid: "e237431e49a34f5ab47d3f33eeef2243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3e3175ddc94b50b362777b9e26511b"
I20240501 20:01:33.374554 26287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10f4d36790d4f628a5cc1c79db3657d" candidate_uuid: "e237431e49a34f5ab47d3f33eeef2243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9884f23a90142ebb9db9bc679df0020"
I20240501 20:01:33.374627 26156 raft_consensus.cc:3037] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:33.374657 26287 raft_consensus.cc:3037] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:33.375537 26287 raft_consensus.cc:2454] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e237431e49a34f5ab47d3f33eeef2243 in term 1.
I20240501 20:01:33.375547 26156 raft_consensus.cc:2454] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e237431e49a34f5ab47d3f33eeef2243 in term 1.
I20240501 20:01:33.375782 25960 leader_election.cc:302] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e237431e49a34f5ab47d3f33eeef2243, f9884f23a90142ebb9db9bc679df0020; no voters:
I20240501 20:01:33.375970 26356 raft_consensus.cc:2781] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:33.376204 26356 raft_consensus.cc:686] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [term 1 LEADER]: Becoming Leader. State: Replica: e237431e49a34f5ab47d3f33eeef2243, State: Running, Role: LEADER
I20240501 20:01:33.376479 26356 consensus_queue.cc:237] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3e3175ddc94b50b362777b9e26511b" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35615 } } peers { permanent_uuid: "e237431e49a34f5ab47d3f33eeef2243" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45351 } } peers { permanent_uuid: "f9884f23a90142ebb9db9bc679df0020" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39639 } }
I20240501 20:01:33.379415 25881 catalog_manager.cc:5484] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 reported cstate change: term changed from 0 to 1, leader changed from <none> to e237431e49a34f5ab47d3f33eeef2243 (127.8.168.65). New cstate: current_term: 1 leader_uuid: "e237431e49a34f5ab47d3f33eeef2243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3e3175ddc94b50b362777b9e26511b" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e237431e49a34f5ab47d3f33eeef2243" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 45351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9884f23a90142ebb9db9bc679df0020" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39639 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:33.419276 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:33.416247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40682 (local address 127.8.168.65:45351)
0501 20:01:33.416370 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:33.416374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:33.416388 (+ 14us) server_negotiation.cc:407] Connection header received
0501 20:01:33.416498 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:33.416501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:33.416575 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:33.416683 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:01:33.417060 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.417588 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:33.418213 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:33.418394 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:33.418488 (+ 94us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:01:33.418904 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:01:33.419011 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:33.419093 (+ 82us) server_negotiation.cc:299] Negotiation successful
0501 20:01:33.419144 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240501 20:01:33.424571 26156 raft_consensus.cc:1261] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b [term 1 FOLLOWER]: Refusing update from remote peer e237431e49a34f5ab47d3f33eeef2243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:33.424571 26287 raft_consensus.cc:1261] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020 [term 1 FOLLOWER]: Refusing update from remote peer e237431e49a34f5ab47d3f33eeef2243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:33.425330 26356 consensus_queue.cc:1035] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9884f23a90142ebb9db9bc679df0020" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:33.425523 26361 consensus_queue.cc:1035] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3e3175ddc94b50b362777b9e26511b" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:33.440109 25881 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58066:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:01:33.440289 25881 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58066:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:01:33.442607 25881 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 914f7701b4714d15a74c3e8ea08b6bef: Sending DeleteTablet for 3 replicas of tablet e10f4d36790d4f628a5cc1c79db3657d
I20240501 20:01:33.443305 26136 tablet_service.cc:1514] Processing DeleteTablet for tablet e10f4d36790d4f628a5cc1c79db3657d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:33 UTC) from {username='slave'} at 127.0.0.1:33654
I20240501 20:01:33.443387 26005 tablet_service.cc:1514] Processing DeleteTablet for tablet e10f4d36790d4f628a5cc1c79db3657d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:33 UTC) from {username='slave'} at 127.0.0.1:40672
I20240501 20:01:33.443804 26375 tablet_replica.cc:330] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243: stopping tablet replica
I20240501 20:01:33.444125 26375 raft_consensus.cc:2229] T e10f4d36790d4f628a5cc1c79db3657d P e237431e49a34f5ab47d3f33eeef2243 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:33.444105 26267 tablet_service.cc:1514] Processing DeleteTablet for tablet e10f4d36790d4f628a5cc1c79db3657d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:33 UTC) from {username='slave'} at 127.0.0.1:48462
I20240501 20:01:33.444618 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 25943
I20240501 20:01:33.445236 26376 tablet_replica.cc:330] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020: stopping tablet replica
I20240501 20:01:33.445456 26376 raft_consensus.cc:2229] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:33.445730 26376 raft_consensus.cc:2258] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:33.446852 26376 ts_tablet_manager.cc:1875] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:01:33.448750 26376 ts_tablet_manager.cc:1888] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:01:33.448837 26376 log.cc:1196] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/e10f4d36790d4f628a5cc1c79db3657d
I20240501 20:01:33.448971 26374 tablet_replica.cc:330] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b: stopping tablet replica
I20240501 20:01:33.449198 26376 ts_tablet_manager.cc:1909] T e10f4d36790d4f628a5cc1c79db3657d P f9884f23a90142ebb9db9bc679df0020: Deleting consensus metadata
I20240501 20:01:33.449199 26374 raft_consensus.cc:2229] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:33.449491 26374 raft_consensus.cc:2258] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 20:01:33.450287 25867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.168.65:45351 (error 108)
I20240501 20:01:33.450274 25869 catalog_manager.cc:4830] TS f9884f23a90142ebb9db9bc679df0020 (127.8.168.67:39639): tablet e10f4d36790d4f628a5cc1c79db3657d (table test-table [id=465c81f6cf5b46a3adcc8b4f2870ac2f]) successfully deleted
I20240501 20:01:33.450784 26374 ts_tablet_manager.cc:1875] T e10f4d36790d4f628a5cc1c79db3657d P bd3e3175ddc94b50b362777b9e26511b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:01:33.451050 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 26074
I20240501 20:01:33.451267 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:33.450963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:45351 (local address 127.0.0.1:40690)
0501 20:01:33.451125 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:33.451181 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20240501 20:01:33.452023 25867 catalog_manager.cc:4573] TS e237431e49a34f5ab47d3f33eeef2243 (127.8.168.65:45351): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e10f4d36790d4f628a5cc1c79db3657d: Network error: Client connection negotiation failed: client connection to 127.8.168.65:45351: connect: Connection refused (error 111)
I20240501 20:01:33.455787 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 26205
I20240501 20:01:33.457154 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:33.455986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:35615 (local address 127.0.0.1:33666)
0501 20:01:33.457017 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:33.457064 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.66:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":951}
I20240501 20:01:33.460755 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 25852
2024-05-01T20:01:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1132 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:33 dist-test-slave-tzct krb5kdc[26380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:33 dist-test-slave-tzct krb5kdc[26380](info): set up 2 sockets
May 01 20:01:33 dist-test-slave-tzct krb5kdc[26380](info): commencing operation
krb5kdc: starting...
W20240501 20:01:34.356673 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.876s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:34 dist-test-slave-tzct krb5kdc[26380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 20:01:34.467597 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:46109
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:46109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:34.576551 26393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:34.576855 26393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:34.576929 26393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:34.580015 26393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:34.580083 26393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:34.580113 26393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:34.580138 26393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:34.580169 26393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:34.584427 26393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:46109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:34.585381 26393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:34.586230 26393 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:34.593272 26399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:34.593374 26398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:34.593770 26401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:34.594349 26393 server_base.cc:1034] running on GCE node
I20240501 20:01:34.594870 26393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:34.595713 26393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:34.596890 26393 hybrid_clock.cc:648] HybridClock initialized: now 1714593694596878 us; error 25 us; skew 500 ppm
May 01 20:01:34 dist-test-slave-tzct krb5kdc[26380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593694, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:34.599442 26393 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:01:34.600333 26393 webserver.cc:460] Webserver started at http://127.8.168.126:35173/ using document root <none> and password file <none>
I20240501 20:01:34.600883 26393 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:34.600950 26393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:34.601106 26393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:34.602888 26393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "53c6926067014532acca33e0b66ddd5c"
format_stamp: "Formatted at 2024-05-01 20:01:34 on dist-test-slave-tzct"
server_key: "f50536543ac2081567a74d24bbd55a04"
server_key_iv: "dc82c358650ee08ba7590172f1b2395d"
server_key_version: "encryptionkey@0"
I20240501 20:01:34.603324 26393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "53c6926067014532acca33e0b66ddd5c"
format_stamp: "Formatted at 2024-05-01 20:01:34 on dist-test-slave-tzct"
server_key: "f50536543ac2081567a74d24bbd55a04"
server_key_iv: "dc82c358650ee08ba7590172f1b2395d"
server_key_version: "encryptionkey@0"
I20240501 20:01:34.606670 26393 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 20:01:34.608935 26408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:34.610117 26393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:34.610249 26393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "53c6926067014532acca33e0b66ddd5c"
format_stamp: "Formatted at 2024-05-01 20:01:34 on dist-test-slave-tzct"
server_key: "f50536543ac2081567a74d24bbd55a04"
server_key_iv: "dc82c358650ee08ba7590172f1b2395d"
server_key_version: "encryptionkey@0"
I20240501 20:01:34.610349 26393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:34.622327 26393 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:34.625535 26393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:34.625695 26393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:34.632426 26393 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:46109
I20240501 20:01:34.632454 26460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:46109 every 8 connection(s)
I20240501 20:01:34.633438 26393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:34.636209 26461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:34.641206 26461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c: Bootstrap starting.
I20240501 20:01:34.642695 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 26393
I20240501 20:01:34.642815 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:34.643096 8865 external_mini_cluster.cc:1428] Setting key df2f1c7e10e8223f4d8d670e91ff702e
I20240501 20:01:34.643487 26461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:34.644202 26461 log.cc:824] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:34.646018 26461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c: No bootstrap required, opened a new log
I20240501 20:01:34.648943 26461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c6926067014532acca33e0b66ddd5c" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 46109 } }
I20240501 20:01:34.649123 26461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:34.649171 26461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c6926067014532acca33e0b66ddd5c, State: Initialized, Role: FOLLOWER
I20240501 20:01:34.649577 26461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c6926067014532acca33e0b66ddd5c" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 46109 } }
I20240501 20:01:34.649701 26461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:34.649770 26461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:34.649878 26461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [term 0 FOLLOWER]: Advancing to term 1
May 01 20:01:34 dist-test-slave-tzct krb5kdc[26380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:01:34.650677 26461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c6926067014532acca33e0b66ddd5c" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 46109 } }
I20240501 20:01:34.650992 26461 leader_election.cc:302] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53c6926067014532acca33e0b66ddd5c; no voters:
I20240501 20:01:34.651301 26461 leader_election.cc:288] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:34.651396 26466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:34.651607 26466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [term 1 LEADER]: Becoming Leader. State: Replica: 53c6926067014532acca33e0b66ddd5c, State: Running, Role: LEADER
I20240501 20:01:34.651966 26466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c6926067014532acca33e0b66ddd5c" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 46109 } }
I20240501 20:01:34.652392 26461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:34.653548 26467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53c6926067014532acca33e0b66ddd5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c6926067014532acca33e0b66ddd5c" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 46109 } } }
I20240501 20:01:34.653673 26467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:34.654094 26468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53c6926067014532acca33e0b66ddd5c. Latest consensus state: current_term: 1 leader_uuid: "53c6926067014532acca33e0b66ddd5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c6926067014532acca33e0b66ddd5c" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 46109 } } }
I20240501 20:01:34.654228 26468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:34.656237 26464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:34.644146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.8.168.126:46109)
0501 20:01:34.644583 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:34.644594 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:34.644636 (+ 42us) server_negotiation.cc:407] Connection header received
0501 20:01:34.645357 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:34.645377 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:34.645656 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:34.645992 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:34.646846 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:34.647611 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:34.648249 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:34.648516 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:34.650712 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:34.650731 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:34.650742 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:34.650775 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:34.652942 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:34.653474 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:34.653480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:34.653486 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:34.653580 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:34.653855 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:34.653858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:34.653860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:34.654332 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:34.654569 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:34.655256 (+ 687us) server_negotiation.cc:299] Negotiation successful
0501 20:01:34.655518 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":151,"threads_started":1}
W20240501 20:01:34.657016 26481 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:01:34.657083 26481 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:01:34.657176 26475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:34.658114 26475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:34.663429 26475 catalog_manager.cc:1347] Generated new cluster ID: 765b1ea3e9b144b19011d661b90576fc
I20240501 20:01:34.663499 26475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:34.669607 26475 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:34.670507 26475 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:34.682684 26475 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c: Generated new TSK 0
I20240501 20:01:34.683395 26475 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
I20240501 20:01:34.713580 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:46109
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:34.828922 26487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:34.829118 26487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:34.829180 26487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:34.832181 26487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:34.832237 26487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:34.832325 26487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 20:01:34.836386 26487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:46109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:34.837250 26487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:34.838057 26487 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:34.846061 26493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:34.846371 26492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:34.846518 26495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:34.847059 26487 server_base.cc:1034] running on GCE node
I20240501 20:01:34.847388 26487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:34.847890 26487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:34.849031 26487 hybrid_clock.cc:648] HybridClock initialized: now 1714593694848971 us; error 74 us; skew 500 ppm
May 01 20:01:34 dist-test-slave-tzct krb5kdc[26380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593694, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:34.851716 26487 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 20:01:34.852663 26487 webserver.cc:460] Webserver started at http://127.8.168.65:41553/ using document root <none> and password file <none>
I20240501 20:01:34.853200 26487 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:34.853253 26487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:34.853395 26487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:34.855177 26487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "1e1ba481fb7a4a7ab2f206a300d3d0b9"
format_stamp: "Formatted at 2024-05-01 20:01:34 on dist-test-slave-tzct"
server_key: "3f312062e81579da723c38e26a8cf1f9"
server_key_iv: "263b3417a690aa139fb658b5a3231700"
server_key_version: "encryptionkey@0"
I20240501 20:01:34.855607 26487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "1e1ba481fb7a4a7ab2f206a300d3d0b9"
format_stamp: "Formatted at 2024-05-01 20:01:34 on dist-test-slave-tzct"
server_key: "3f312062e81579da723c38e26a8cf1f9"
server_key_iv: "263b3417a690aa139fb658b5a3231700"
server_key_version: "encryptionkey@0"
I20240501 20:01:34.858762 26487 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240501 20:01:34.860936 26502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:34.861934 26487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:34.862037 26487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "1e1ba481fb7a4a7ab2f206a300d3d0b9"
format_stamp: "Formatted at 2024-05-01 20:01:34 on dist-test-slave-tzct"
server_key: "3f312062e81579da723c38e26a8cf1f9"
server_key_iv: "263b3417a690aa139fb658b5a3231700"
server_key_version: "encryptionkey@0"
I20240501 20:01:34.862128 26487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:34.881013 26487 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:34.883708 26487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:34.883874 26487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:34.884415 26487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:34.885283 26487 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:34.885331 26487 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:34.885377 26487 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:34.885404 26487 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:34.893579 26487 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:36335
I20240501 20:01:34.893606 26615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:36335 every 8 connection(s)
I20240501 20:01:34.894510 26487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 20:01:34.899911 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 26487
I20240501 20:01:34.900043 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 20:01:34.900305 8865 external_mini_cluster.cc:1428] Setting key 151b0a48c23f53f0581612c840a6dbd3
May 01 20:01:34 dist-test-slave-tzct krb5kdc[26380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593694, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:01:34.906744 26464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:34.896280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:51559 (local address 127.8.168.126:46109)
0501 20:01:34.896411 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:34.896415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:34.897063 (+ 648us) server_negotiation.cc:407] Connection header received
0501 20:01:34.897837 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:34.897841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:34.897894 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:34.897971 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:34.899514 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:34.900048 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:34.900856 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:34.901054 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:34.903382 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:34.903401 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:34.903404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:34.903429 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:34.904805 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:34.905342 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:34.905346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:34.905348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:34.905395 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:34.905662 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:34.905666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:34.905668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:34.905840 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:34.905936 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:34.906442 (+ 506us) server_negotiation.cc:299] Negotiation successful
0501 20:01:34.906576 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240501 20:01:34.907398 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:34.896537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:46109 (local address 127.8.168.65:51559)
0501 20:01:34.896937 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:34.896971 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:34.897653 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:34.898122 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:34.898129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:34.898507 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:34.899383 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:34.899397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:34.900164 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:34.900168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:34.900704 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:34.900713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:34.900926 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:34.901606 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:01:34.901634 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:34.903205 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:34.904915 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:34.904921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:34.904933 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:34.905245 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:34.905472 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:34.905476 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:34.905478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:34.905582 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:34.905948 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:34.905954 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:34.906205 (+ 251us) client_negotiation.cc:769] Sending connection context
0501 20:01:34.906389 (+ 184us) client_negotiation.cc:240] Negotiation successful
0501 20:01:34.906673 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":133,"threads_started":1}
WARNING: no policy specified for kudu/127.8.168.66@KRBTEST.COM; defaulting to no policy
I20240501 20:01:34.908648 26616 heartbeater.cc:347] Connected to a master server at 127.8.168.126:46109
I20240501 20:01:34.908921 26616 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:34.909430 26616 heartbeater.cc:511] Master 127.8.168.126:46109 requested a full tablet report, sending...
I20240501 20:01:34.910960 26425 ts_manager.cc:194] Registered new tserver with Master: 1e1ba481fb7a4a7ab2f206a300d3d0b9 (127.8.168.65:36335)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.66@KRBTEST.COM" created.
I20240501 20:01:34.912202 26425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:51559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
I20240501 20:01:34.927462 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:46109
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:35.030988 26621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:35.031203 26621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:35.031261 26621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:35.034336 26621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:35.034396 26621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:35.034490 26621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 20:01:35.038570 26621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:46109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:35.039388 26621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:35.040187 26621 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:35.047806 26629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:35.048171 26626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:35.048614 26627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:35.049083 26621 server_base.cc:1034] running on GCE node
I20240501 20:01:35.049439 26621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:35.050025 26621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:35.051191 26621 hybrid_clock.cc:648] HybridClock initialized: now 1714593695051170 us; error 39 us; skew 500 ppm
May 01 20:01:35 dist-test-slave-tzct krb5kdc[26380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593695, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:35.053820 26621 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 20:01:35.054747 26621 webserver.cc:460] Webserver started at http://127.8.168.66:35865/ using document root <none> and password file <none>
I20240501 20:01:35.055269 26621 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:35.055325 26621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:35.055496 26621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:35.057297 26621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "eabe8faafa024b5eb0985a5832365d14"
format_stamp: "Formatted at 2024-05-01 20:01:35 on dist-test-slave-tzct"
server_key: "1396fd56ed64e559095c2a0172a3f97d"
server_key_iv: "2ff93916a8a2869330d5b870ecd9b9dc"
server_key_version: "encryptionkey@0"
I20240501 20:01:35.057751 26621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "eabe8faafa024b5eb0985a5832365d14"
format_stamp: "Formatted at 2024-05-01 20:01:35 on dist-test-slave-tzct"
server_key: "1396fd56ed64e559095c2a0172a3f97d"
server_key_iv: "2ff93916a8a2869330d5b870ecd9b9dc"
server_key_version: "encryptionkey@0"
I20240501 20:01:35.061038 26621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:01:35.063302 26636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:35.064321 26621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:35.064422 26621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "eabe8faafa024b5eb0985a5832365d14"
format_stamp: "Formatted at 2024-05-01 20:01:35 on dist-test-slave-tzct"
server_key: "1396fd56ed64e559095c2a0172a3f97d"
server_key_iv: "2ff93916a8a2869330d5b870ecd9b9dc"
server_key_version: "encryptionkey@0"
I20240501 20:01:35.064512 26621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:35.076781 26621 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:35.079854 26621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:35.080019 26621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:35.080559 26621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:35.081372 26621 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:35.081419 26621 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:35.081459 26621 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:35.081486 26621 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:35.089459 26621 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:46477
I20240501 20:01:35.089504 26749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:46477 every 8 connection(s)
I20240501 20:01:35.090353 26621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 20:01:35.092765 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 26621
I20240501 20:01:35.092902 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 20:01:35.093175 8865 external_mini_cluster.cc:1428] Setting key 39bcd77cc74ecf732376002b5889d357
May 01 20:01:35 dist-test-slave-tzct krb5kdc[26380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593695, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.168.67@KRBTEST.COM; defaulting to no policy
I20240501 20:01:35.102983 26464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.091964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:37677 (local address 127.8.168.126:46109)
0501 20:01:35.092134 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:35.092139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:35.092929 (+ 790us) server_negotiation.cc:407] Connection header received
0501 20:01:35.093878 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:35.093882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:35.093939 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:35.094036 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:35.096016 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.096703 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:35.097333 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.097516 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:35.099712 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:35.099733 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:35.099737 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:35.099768 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:35.101289 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:35.101735 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:35.101739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:35.101741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:35.101797 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:35.102021 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:35.102024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:35.102026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:35.102218 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:35.102327 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:35.102753 (+ 426us) server_negotiation.cc:299] Negotiation successful
0501 20:01:35.102856 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20240501 20:01:35.103580 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.092238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:46109 (local address 127.8.168.66:37677)
0501 20:01:35.092683 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:35.092761 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:35.093619 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:35.094164 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:35.094172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:35.094662 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:35.095848 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:35.095866 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.096839 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:35.096841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:35.097224 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:35.097231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.097394 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:35.097951 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:01:35.097975 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:35.099556 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:35.101407 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:35.101411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:35.101420 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:35.101641 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:35.101878 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:35.101880 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:35.101882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:35.101950 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:35.102313 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:35.102317 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:35.102536 (+ 219us) client_negotiation.cc:769] Sending connection context
0501 20:01:35.102699 (+ 163us) client_negotiation.cc:240] Negotiation successful
0501 20:01:35.102915 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":95,"threads_started":1}
I20240501 20:01:35.104676 26750 heartbeater.cc:347] Connected to a master server at 127.8.168.126:46109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.67@KRBTEST.COM" created.
I20240501 20:01:35.104948 26750 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:35.105453 26750 heartbeater.cc:511] Master 127.8.168.126:46109 requested a full tablet report, sending...
I20240501 20:01:35.106467 26425 ts_manager.cc:194] Registered new tserver with Master: eabe8faafa024b5eb0985a5832365d14 (127.8.168.66:46477)
I20240501 20:01:35.107013 26425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:37677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
I20240501 20:01:35.119696 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:46109
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:35.226918 26755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:35.227160 26755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:35.227221 26755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:35.230382 26755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:35.230459 26755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:35.230558 26755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 20:01:35.234871 26755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:46109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:35.235805 26755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:35.236642 26755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:35.244639 26763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:35.244860 26760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:35.245350 26761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:35.245790 26755 server_base.cc:1034] running on GCE node
I20240501 20:01:35.246168 26755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:35.246760 26755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:35.247938 26755 hybrid_clock.cc:648] HybridClock initialized: now 1714593695247924 us; error 29 us; skew 500 ppm
May 01 20:01:35 dist-test-slave-tzct krb5kdc[26380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593695, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:35.255148 26755 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 20:01:35.256155 26755 webserver.cc:460] Webserver started at http://127.8.168.67:42121/ using document root <none> and password file <none>
I20240501 20:01:35.256685 26755 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:35.256772 26755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:35.256947 26755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:35.258744 26755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "b10b664bf215400f9ea2b1922049b142"
format_stamp: "Formatted at 2024-05-01 20:01:35 on dist-test-slave-tzct"
server_key: "dbf1d07c50cc31b266090b0d766892ff"
server_key_iv: "3d7ec1264fa085e3640d51a817e4c9a3"
server_key_version: "encryptionkey@0"
I20240501 20:01:35.259209 26755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "b10b664bf215400f9ea2b1922049b142"
format_stamp: "Formatted at 2024-05-01 20:01:35 on dist-test-slave-tzct"
server_key: "dbf1d07c50cc31b266090b0d766892ff"
server_key_iv: "3d7ec1264fa085e3640d51a817e4c9a3"
server_key_version: "encryptionkey@0"
I20240501 20:01:35.262697 26755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240501 20:01:35.265064 26770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:35.266191 26755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:01:35.266302 26755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "b10b664bf215400f9ea2b1922049b142"
format_stamp: "Formatted at 2024-05-01 20:01:35 on dist-test-slave-tzct"
server_key: "dbf1d07c50cc31b266090b0d766892ff"
server_key_iv: "3d7ec1264fa085e3640d51a817e4c9a3"
server_key_version: "encryptionkey@0"
I20240501 20:01:35.266392 26755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:35.282053 26755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:35.284834 26755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:35.284999 26755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:35.285523 26755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:35.286384 26755 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:35.286434 26755 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:35.286473 26755 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:35.286494 26755 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:35.294833 26755 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:39631
I20240501 20:01:35.294865 26883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:39631 every 8 connection(s)
I20240501 20:01:35.295810 26755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
May 01 20:01:35 dist-test-slave-tzct krb5kdc[26380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593695, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:01:35.304927 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 26755
I20240501 20:01:35.305056 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 20:01:35.305318 8865 external_mini_cluster.cc:1428] Setting key f1dbfa567ae61b984c2321275c42b8d5
I20240501 20:01:35.308334 26464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.297577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:37711 (local address 127.8.168.126:46109)
0501 20:01:35.297704 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:35.297708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:35.298324 (+ 616us) server_negotiation.cc:407] Connection header received
0501 20:01:35.299151 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:35.299155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:35.299218 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:35.299292 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:35.301027 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.301534 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:35.302193 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.302374 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:35.304635 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:35.304660 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:35.304662 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:35.304691 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:35.306423 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:35.306947 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:35.306951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:35.306954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:35.307007 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:35.307265 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:35.307269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:35.307271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:35.307436 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:35.307529 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:35.308050 (+ 521us) server_negotiation.cc:299] Negotiation successful
0501 20:01:35.308175 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240501 20:01:35.309043 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.297766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:46109 (local address 127.8.168.67:37711)
0501 20:01:35.298181 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:35.298218 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:35.298937 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:35.299468 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:35.299477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:35.299871 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:35.300861 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:35.300879 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.301661 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:35.301664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:35.302071 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:35.302079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.302304 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:35.302956 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:01:35.302987 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:35.304457 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:35.306534 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:35.306540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:35.306551 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:35.306860 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:35.307083 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:35.307086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:35.307089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:35.307185 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:35.307532 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:35.307539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:35.307799 (+ 260us) client_negotiation.cc:769] Sending connection context
0501 20:01:35.307993 (+ 194us) client_negotiation.cc:240] Negotiation successful
0501 20:01:35.308285 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20240501 20:01:35.310110 26884 heartbeater.cc:347] Connected to a master server at 127.8.168.126:46109
I20240501 20:01:35.310380 26884 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:35.311029 26884 heartbeater.cc:511] Master 127.8.168.126:46109 requested a full tablet report, sending...
I20240501 20:01:35.311996 26425 ts_manager.cc:194] Registered new tserver with Master: b10b664bf215400f9ea2b1922049b142 (127.8.168.67:39631)
I20240501 20:01:35.312565 26425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:37711
I20240501 20:01:35.319320 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 20:01:35.327419 26464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.321110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.8.168.126:46109)
0501 20:01:35.321255 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:35.321260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:35.321370 (+ 110us) server_negotiation.cc:407] Connection header received
0501 20:01:35.321483 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:35.321495 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:35.321537 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:35.321603 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:35.322325 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.322817 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:35.323458 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.323581 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:35.324833 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:35.324849 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:35.324851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:35.324870 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:35.325995 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:35.326386 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:35.326388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:35.326389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:35.326426 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:35.326714 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:35.326716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:35.326717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:35.326867 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:35.326982 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:35.327186 (+ 204us) server_negotiation.cc:299] Negotiation successful
0501 20:01:35.327258 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20240501 20:01:35.330238 26425 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:01:35.332674 26425 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:01:35.345613 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.341306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:46477 (local address 127.0.0.1:36636)
0501 20:01:35.341992 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:35.342012 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:35.342149 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:35.343094 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:35.343097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:35.343112 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:35.343305 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:35.343312 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.344226 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:35.344229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:35.345072 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:35.345081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.345183 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:35.345250 (+ 67us) client_negotiation.cc:769] Sending connection context
0501 20:01:35.345352 (+ 102us) client_negotiation.cc:240] Negotiation successful
0501 20:01:35.345447 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"spinlock_wait_cycles":1792,"thread_start_us":140,"threads_started":1}
I20240501 20:01:35.345862 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.341040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:39631 (local address 127.0.0.1:37872)
0501 20:01:35.342366 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:35.342405 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:35.342484 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:35.342908 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:35.342911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:35.342929 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:35.343156 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:35.343164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.344543 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:35.344546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:35.345477 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:35.345487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.345589 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:35.345606 (+ 17us) client_negotiation.cc:769] Sending connection context
0501 20:01:35.345652 (+ 46us) client_negotiation.cc:240] Negotiation successful
0501 20:01:35.345711 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1260,"mutex_wait_us":576,"thread_start_us":99,"threads_started":1}
I20240501 20:01:35.346381 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.342016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36636 (local address 127.8.168.66:46477)
0501 20:01:35.342686 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:35.342692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:35.342713 (+ 21us) server_negotiation.cc:407] Connection header received
0501 20:01:35.342793 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:35.342798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:35.342957 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:35.343104 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:35.343419 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.344123 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:35.345469 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.345968 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:35.346098 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:35.346172 (+ 74us) server_negotiation.cc:299] Negotiation successful
0501 20:01:35.346258 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":119,"threads_started":1}
I20240501 20:01:35.346918 26897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.341261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.8.168.67:39631)
0501 20:01:35.341885 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:35.341891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:35.342482 (+ 591us) server_negotiation.cc:407] Connection header received
0501 20:01:35.342591 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:35.342596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:35.342737 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:35.342874 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:35.343326 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.344434 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:35.346055 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.346569 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:35.346653 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:35.346710 (+ 57us) server_negotiation.cc:299] Negotiation successful
0501 20:01:35.346782 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":140,"threads_started":1}
I20240501 20:01:35.346921 26896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.341086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.8.168.65:36335)
0501 20:01:35.341786 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:35.341793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:35.342288 (+ 495us) server_negotiation.cc:407] Connection header received
0501 20:01:35.342393 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:35.342397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:35.342578 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:35.342737 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:35.343233 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.344397 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:35.345950 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.346482 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:35.346597 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:35.346673 (+ 76us) server_negotiation.cc:299] Negotiation successful
0501 20:01:35.346763 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":160,"threads_started":1}
I20240501 20:01:35.345863 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.341033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:36335 (local address 127.0.0.1:32856)
0501 20:01:35.341973 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:35.342012 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:35.342149 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:35.342768 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:35.342774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:35.342796 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:35.343102 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:35.343111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.344738 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:35.344741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:35.345553 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:35.345561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.345641 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:35.345653 (+ 12us) client_negotiation.cc:769] Sending connection context
0501 20:01:35.345695 (+ 42us) client_negotiation.cc:240] Negotiation successful
0501 20:01:35.345738 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"spinlock_wait_cycles":3840,"thread_start_us":101,"threads_started":1}
I20240501 20:01:35.348795 26684 tablet_service.cc:1467] Processing CreateTablet for tablet a9557e783c8e4f6e9c4c8eccc0e33751 (DEFAULT_TABLE table=test-table [id=1dc631d8b2cd4ee7884337a03955522d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:35.349351 26550 tablet_service.cc:1467] Processing CreateTablet for tablet a9557e783c8e4f6e9c4c8eccc0e33751 (DEFAULT_TABLE table=test-table [id=1dc631d8b2cd4ee7884337a03955522d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:35.349651 26684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9557e783c8e4f6e9c4c8eccc0e33751. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:35.350076 26550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9557e783c8e4f6e9c4c8eccc0e33751. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:35.355248 26902 tablet_bootstrap.cc:492] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9: Bootstrap starting.
I20240501 20:01:35.355625 26818 tablet_service.cc:1467] Processing CreateTablet for tablet a9557e783c8e4f6e9c4c8eccc0e33751 (DEFAULT_TABLE table=test-table [id=1dc631d8b2cd4ee7884337a03955522d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:35.356209 26901 tablet_bootstrap.cc:492] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14: Bootstrap starting.
I20240501 20:01:35.356581 26818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9557e783c8e4f6e9c4c8eccc0e33751. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:35.357417 26902 tablet_bootstrap.cc:654] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:35.358083 26901 tablet_bootstrap.cc:654] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:35.358749 26902 log.cc:824] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:35.358877 26901 log.cc:824] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:35.360399 26901 tablet_bootstrap.cc:492] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14: No bootstrap required, opened a new log
I20240501 20:01:35.360520 26902 tablet_bootstrap.cc:492] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9: No bootstrap required, opened a new log
I20240501 20:01:35.360697 26901 ts_tablet_manager.cc:1367] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:01:35.360703 26902 ts_tablet_manager.cc:1367] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:01:35.363384 26905 tablet_bootstrap.cc:492] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142: Bootstrap starting.
I20240501 20:01:35.363636 26901 raft_consensus.cc:348] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ba481fb7a4a7ab2f206a300d3d0b9" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 36335 } } peers { permanent_uuid: "b10b664bf215400f9ea2b1922049b142" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39631 } } peers { permanent_uuid: "eabe8faafa024b5eb0985a5832365d14" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 46477 } }
I20240501 20:01:35.363814 26901 raft_consensus.cc:374] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:35.363893 26901 raft_consensus.cc:729] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eabe8faafa024b5eb0985a5832365d14, State: Initialized, Role: FOLLOWER
I20240501 20:01:35.364301 26901 consensus_queue.cc:260] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ba481fb7a4a7ab2f206a300d3d0b9" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 36335 } } peers { permanent_uuid: "b10b664bf215400f9ea2b1922049b142" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39631 } } peers { permanent_uuid: "eabe8faafa024b5eb0985a5832365d14" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 46477 } }
I20240501 20:01:35.364271 26902 raft_consensus.cc:348] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ba481fb7a4a7ab2f206a300d3d0b9" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 36335 } } peers { permanent_uuid: "b10b664bf215400f9ea2b1922049b142" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39631 } } peers { permanent_uuid: "eabe8faafa024b5eb0985a5832365d14" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 46477 } }
I20240501 20:01:35.364513 26902 raft_consensus.cc:374] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:35.364578 26902 raft_consensus.cc:729] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e1ba481fb7a4a7ab2f206a300d3d0b9, State: Initialized, Role: FOLLOWER
I20240501 20:01:35.365057 26902 consensus_queue.cc:260] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ba481fb7a4a7ab2f206a300d3d0b9" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 36335 } } peers { permanent_uuid: "b10b664bf215400f9ea2b1922049b142" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39631 } } peers { permanent_uuid: "eabe8faafa024b5eb0985a5832365d14" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 46477 } }
I20240501 20:01:35.365159 26901 ts_tablet_manager.cc:1398] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:01:35.365506 26750 heartbeater.cc:503] Master 127.8.168.126:46109 was elected leader, sending a full tablet report...
I20240501 20:01:35.365581 26905 tablet_bootstrap.cc:654] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:35.365641 26616 heartbeater.cc:503] Master 127.8.168.126:46109 was elected leader, sending a full tablet report...
I20240501 20:01:35.365918 26902 ts_tablet_manager.cc:1398] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20240501 20:01:35.366392 26905 log.cc:824] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:35.368084 26905 tablet_bootstrap.cc:492] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142: No bootstrap required, opened a new log
I20240501 20:01:35.368249 26905 ts_tablet_manager.cc:1367] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:01:35.371412 26905 raft_consensus.cc:348] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ba481fb7a4a7ab2f206a300d3d0b9" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 36335 } } peers { permanent_uuid: "b10b664bf215400f9ea2b1922049b142" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39631 } } peers { permanent_uuid: "eabe8faafa024b5eb0985a5832365d14" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 46477 } }
I20240501 20:01:35.371592 26905 raft_consensus.cc:374] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:35.371636 26905 raft_consensus.cc:729] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b10b664bf215400f9ea2b1922049b142, State: Initialized, Role: FOLLOWER
I20240501 20:01:35.372007 26905 consensus_queue.cc:260] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ba481fb7a4a7ab2f206a300d3d0b9" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 36335 } } peers { permanent_uuid: "b10b664bf215400f9ea2b1922049b142" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39631 } } peers { permanent_uuid: "eabe8faafa024b5eb0985a5832365d14" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 46477 } }
I20240501 20:01:35.372682 26884 heartbeater.cc:503] Master 127.8.168.126:46109 was elected leader, sending a full tablet report...
I20240501 20:01:35.372767 26905 ts_tablet_manager.cc:1398] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
W20240501 20:01:35.395674 26617 tablet.cc:2369] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:01:35.546744 26885 tablet.cc:2369] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:01:35.576427 26906 raft_consensus.cc:482] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:35.576635 26906 raft_consensus.cc:504] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ba481fb7a4a7ab2f206a300d3d0b9" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 36335 } } peers { permanent_uuid: "b10b664bf215400f9ea2b1922049b142" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39631 } } peers { permanent_uuid: "eabe8faafa024b5eb0985a5832365d14" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 46477 } }
I20240501 20:01:35.577808 26906 leader_election.cc:288] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e1ba481fb7a4a7ab2f206a300d3d0b9 (127.8.168.65:36335), b10b664bf215400f9ea2b1922049b142 (127.8.168.67:39631)
I20240501 20:01:35.581446 26910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.578094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:36335 (local address 127.8.168.66:44757)
0501 20:01:35.578433 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:35.578445 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:35.578538 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:35.578788 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:35.578791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:35.578804 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:35.579041 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:35.579048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.580076 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:35.580078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:35.581019 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:35.581032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.581149 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:35.581171 (+ 22us) client_negotiation.cc:769] Sending connection context
0501 20:01:35.581221 (+ 50us) client_negotiation.cc:240] Negotiation successful
0501 20:01:35.581279 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":13,"thread_start_us":108,"threads_started":1}
I20240501 20:01:35.581915 26896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.578209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:44757 (local address 127.8.168.65:36335)
0501 20:01:35.578355 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:35.578360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:35.578514 (+ 154us) server_negotiation.cc:407] Connection header received
0501 20:01:35.578618 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:35.578622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:35.578683 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:35.578764 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:35.579143 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.579944 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:35.581159 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.581640 (+ 481us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:35.581680 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:35.581734 (+ 54us) server_negotiation.cc:299] Negotiation successful
0501 20:01:35.581794 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20240501 20:01:35.582479 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9557e783c8e4f6e9c4c8eccc0e33751" candidate_uuid: "eabe8faafa024b5eb0985a5832365d14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e1ba481fb7a4a7ab2f206a300d3d0b9" is_pre_election: true
I20240501 20:01:35.582785 26570 raft_consensus.cc:2454] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eabe8faafa024b5eb0985a5832365d14 in term 0.
I20240501 20:01:35.582845 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.578084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:39631 (local address 127.8.168.66:36577)
0501 20:01:35.578289 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:35.578304 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:35.578400 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:35.578672 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:35.578674 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:35.578688 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:35.578960 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:35.578967 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.580047 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:35.580050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:35.581043 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:35.581049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.581148 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:35.581171 (+ 23us) client_negotiation.cc:769] Sending connection context
0501 20:01:35.581224 (+ 53us) client_negotiation.cc:240] Negotiation successful
0501 20:01:35.581279 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20240501 20:01:35.583034 26897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.578164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:36577 (local address 127.8.168.67:39631)
0501 20:01:35.578291 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:35.578295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:35.578409 (+ 114us) server_negotiation.cc:407] Connection header received
0501 20:01:35.578481 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:35.578484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:35.578558 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:35.578647 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:35.579079 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.579920 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:35.582014 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.582645 (+ 631us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:35.582690 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:35.582752 (+ 62us) server_negotiation.cc:299] Negotiation successful
0501 20:01:35.582909 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20240501 20:01:35.583251 26639 leader_election.cc:302] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e1ba481fb7a4a7ab2f206a300d3d0b9, eabe8faafa024b5eb0985a5832365d14; no voters:
I20240501 20:01:35.583523 26906 raft_consensus.cc:2781] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:35.583597 26906 raft_consensus.cc:482] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:35.583640 26906 raft_consensus.cc:3037] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:35.583590 26838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9557e783c8e4f6e9c4c8eccc0e33751" candidate_uuid: "eabe8faafa024b5eb0985a5832365d14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b10b664bf215400f9ea2b1922049b142" is_pre_election: true
I20240501 20:01:35.583829 26838 raft_consensus.cc:2454] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eabe8faafa024b5eb0985a5832365d14 in term 0.
I20240501 20:01:35.584664 26906 raft_consensus.cc:504] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ba481fb7a4a7ab2f206a300d3d0b9" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 36335 } } peers { permanent_uuid: "b10b664bf215400f9ea2b1922049b142" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39631 } } peers { permanent_uuid: "eabe8faafa024b5eb0985a5832365d14" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 46477 } }
I20240501 20:01:35.585039 26906 leader_election.cc:288] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [CANDIDATE]: Term 1 election: Requested vote from peers 1e1ba481fb7a4a7ab2f206a300d3d0b9 (127.8.168.65:36335), b10b664bf215400f9ea2b1922049b142 (127.8.168.67:39631)
I20240501 20:01:35.585355 26838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9557e783c8e4f6e9c4c8eccc0e33751" candidate_uuid: "eabe8faafa024b5eb0985a5832365d14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b10b664bf215400f9ea2b1922049b142"
I20240501 20:01:35.585372 26570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9557e783c8e4f6e9c4c8eccc0e33751" candidate_uuid: "eabe8faafa024b5eb0985a5832365d14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e1ba481fb7a4a7ab2f206a300d3d0b9"
I20240501 20:01:35.585464 26838 raft_consensus.cc:3037] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:35.585479 26570 raft_consensus.cc:3037] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:35.586351 26838 raft_consensus.cc:2454] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eabe8faafa024b5eb0985a5832365d14 in term 1.
I20240501 20:01:35.586403 26570 raft_consensus.cc:2454] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eabe8faafa024b5eb0985a5832365d14 in term 1.
I20240501 20:01:35.586671 26637 leader_election.cc:302] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b10b664bf215400f9ea2b1922049b142, eabe8faafa024b5eb0985a5832365d14; no voters:
I20240501 20:01:35.586892 26906 raft_consensus.cc:2781] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:35.587114 26906 raft_consensus.cc:686] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [term 1 LEADER]: Becoming Leader. State: Replica: eabe8faafa024b5eb0985a5832365d14, State: Running, Role: LEADER
I20240501 20:01:35.587430 26906 consensus_queue.cc:237] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ba481fb7a4a7ab2f206a300d3d0b9" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 36335 } } peers { permanent_uuid: "b10b664bf215400f9ea2b1922049b142" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39631 } } peers { permanent_uuid: "eabe8faafa024b5eb0985a5832365d14" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 46477 } }
I20240501 20:01:35.590440 26424 catalog_manager.cc:5484] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 reported cstate change: term changed from 0 to 1, leader changed from <none> to eabe8faafa024b5eb0985a5832365d14 (127.8.168.66). New cstate: current_term: 1 leader_uuid: "eabe8faafa024b5eb0985a5832365d14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e1ba481fb7a4a7ab2f206a300d3d0b9" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 36335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b10b664bf215400f9ea2b1922049b142" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eabe8faafa024b5eb0985a5832365d14" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 46477 } health_report { overall_health: HEALTHY } } }
W20240501 20:01:35.591814 26751 tablet.cc:2369] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:01:35.648320 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.645018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36640 (local address 127.8.168.66:46477)
0501 20:01:35.645148 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:35.645151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:35.645165 (+ 14us) server_negotiation.cc:407] Connection header received
0501 20:01:35.645378 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:35.645382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:35.645447 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:35.645554 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:01:35.645942 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.646498 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:35.647143 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.647279 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:35.647499 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:01:35.647925 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:01:35.648042 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:35.648151 (+ 109us) server_negotiation.cc:299] Negotiation successful
0501 20:01:35.648183 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20240501 20:01:35.653890 26838 raft_consensus.cc:1261] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142 [term 1 FOLLOWER]: Refusing update from remote peer eabe8faafa024b5eb0985a5832365d14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:35.653923 26570 raft_consensus.cc:1261] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9 [term 1 FOLLOWER]: Refusing update from remote peer eabe8faafa024b5eb0985a5832365d14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:35.654621 26906 consensus_queue.cc:1035] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b10b664bf215400f9ea2b1922049b142" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 39631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:35.654805 26911 consensus_queue.cc:1035] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e1ba481fb7a4a7ab2f206a300d3d0b9" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 36335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:35.663168 26916 mvcc.cc:205] Tried to move back new op lower bound from 7022975777390710784 to 7022975777128910848. Current Snapshot: MvccSnapshot[applied={T|T < 7022975777390710784}]
I20240501 20:01:35.671840 26425 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53962:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:01:35.672031 26425 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53962:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:01:35.674352 26425 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 53c6926067014532acca33e0b66ddd5c: Sending DeleteTablet for 3 replicas of tablet a9557e783c8e4f6e9c4c8eccc0e33751
I20240501 20:01:35.675123 26550 tablet_service.cc:1514] Processing DeleteTablet for tablet a9557e783c8e4f6e9c4c8eccc0e33751 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:35 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:32856
I20240501 20:01:35.675122 26684 tablet_service.cc:1514] Processing DeleteTablet for tablet a9557e783c8e4f6e9c4c8eccc0e33751 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:35 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:36636
I20240501 20:01:35.675637 26925 tablet_replica.cc:330] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14: stopping tablet replica
I20240501 20:01:35.675618 26818 tablet_service.cc:1514] Processing DeleteTablet for tablet a9557e783c8e4f6e9c4c8eccc0e33751 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:35 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:37872
I20240501 20:01:35.675899 26924 tablet_replica.cc:330] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9: stopping tablet replica
I20240501 20:01:35.676095 26926 tablet_replica.cc:330] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142: stopping tablet replica
I20240501 20:01:35.676172 26924 raft_consensus.cc:2229] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:35.676322 26926 raft_consensus.cc:2229] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:35.676502 26924 raft_consensus.cc:2258] T a9557e783c8e4f6e9c4c8eccc0e33751 P 1e1ba481fb7a4a7ab2f206a300d3d0b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:35.676707 26926 raft_consensus.cc:2258] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:35.676740 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 26487
I20240501 20:01:35.677208 26925 raft_consensus.cc:2229] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:35.677613 26925 raft_consensus.cc:2258] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:35.678859 26926 ts_tablet_manager.cc:1875] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:01:35.679723 26925 ts_tablet_manager.cc:1875] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:01:35.681707 26926 ts_tablet_manager.cc:1888] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:01:35.681833 26926 log.cc:1196] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/a9557e783c8e4f6e9c4c8eccc0e33751
I20240501 20:01:35.682160 26926 ts_tablet_manager.cc:1909] T a9557e783c8e4f6e9c4c8eccc0e33751 P b10b664bf215400f9ea2b1922049b142: Deleting consensus metadata
I20240501 20:01:35.682548 26925 ts_tablet_manager.cc:1888] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:01:35.682699 26925 log.cc:1196] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/a9557e783c8e4f6e9c4c8eccc0e33751
I20240501 20:01:35.683051 26925 ts_tablet_manager.cc:1909] T a9557e783c8e4f6e9c4c8eccc0e33751 P eabe8faafa024b5eb0985a5832365d14: Deleting consensus metadata
I20240501 20:01:35.683372 26409 catalog_manager.cc:4830] TS b10b664bf215400f9ea2b1922049b142 (127.8.168.67:39631): tablet a9557e783c8e4f6e9c4c8eccc0e33751 (table test-table [id=1dc631d8b2cd4ee7884337a03955522d]) successfully deleted
I20240501 20:01:35.684345 26412 catalog_manager.cc:4830] TS eabe8faafa024b5eb0985a5832365d14 (127.8.168.66:46477): tablet a9557e783c8e4f6e9c4c8eccc0e33751 (table test-table [id=1dc631d8b2cd4ee7884337a03955522d]) successfully deleted
W20240501 20:01:35.685052 26411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.168.65:36335 (error 108)
I20240501 20:01:35.685210 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 26621
I20240501 20:01:35.687166 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.686851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:36335 (local address 127.0.0.1:32866)
0501 20:01:35.687002 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:35.687076 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:36335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20240501 20:01:35.687309 26411 catalog_manager.cc:4573] TS 1e1ba481fb7a4a7ab2f206a300d3d0b9 (127.8.168.65:36335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9557e783c8e4f6e9c4c8eccc0e33751: Network error: Client connection negotiation failed: client connection to 127.8.168.65:36335: connect: Connection refused (error 111)
I20240501 20:01:35.690079 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 26755
I20240501 20:01:35.696072 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 26393
2024-05-01T20:01:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2237 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2024-05-01T20:01:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:35Z Disabled control of system clock
I20240501 20:01:35.724925 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:45745
--webserver_interface=127.8.168.126
--webserver_port=0
--builtin_ntp_servers=127.8.168.84:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:45745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240501 20:01:35.829325 26931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:35.829569 26931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:35.829615 26931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:35.832646 26931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:35.832713 26931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:35.832744 26931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:35.832762 26931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:35.832778 26931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:35.836438 26931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:45745
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:45745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:35.837313 26931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:35.838372 26931 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:35.845036 26937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:35.845402 26936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:35.845774 26939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:35.846215 26931 server_base.cc:1034] running on GCE node
I20240501 20:01:35.846706 26931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:35.847568 26931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:35.848749 26931 hybrid_clock.cc:648] HybridClock initialized: now 1714593695848704 us; error 31 us; skew 500 ppm
I20240501 20:01:35.850620 26931 webserver.cc:460] Webserver started at http://127.8.168.126:38399/ using document root <none> and password file <none>
I20240501 20:01:35.851151 26931 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:35.851208 26931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:35.851375 26931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:35.853173 26931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "61370bad7b034707b525feb5f516f84a"
format_stamp: "Formatted at 2024-05-01 20:01:35 on dist-test-slave-tzct"
server_key: "0ef33fc918de7bbcb2b1062678207fce"
server_key_iv: "c4d3b073eede32d093233daded3b3698"
server_key_version: "encryptionkey@0"
I20240501 20:01:35.853621 26931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "61370bad7b034707b525feb5f516f84a"
format_stamp: "Formatted at 2024-05-01 20:01:35 on dist-test-slave-tzct"
server_key: "0ef33fc918de7bbcb2b1062678207fce"
server_key_iv: "c4d3b073eede32d093233daded3b3698"
server_key_version: "encryptionkey@0"
I20240501 20:01:35.856942 26931 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:01:35.859155 26945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:35.860152 26931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:35.860250 26931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "61370bad7b034707b525feb5f516f84a"
format_stamp: "Formatted at 2024-05-01 20:01:35 on dist-test-slave-tzct"
server_key: "0ef33fc918de7bbcb2b1062678207fce"
server_key_iv: "c4d3b073eede32d093233daded3b3698"
server_key_version: "encryptionkey@0"
I20240501 20:01:35.860330 26931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:35.882863 26931 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:35.883529 26931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:35.883674 26931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:35.890192 26931 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:45745
I20240501 20:01:35.890199 26997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:45745 every 8 connection(s)
I20240501 20:01:35.891184 26931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:35.893822 26998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:35.899144 26998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a: Bootstrap starting.
I20240501 20:01:35.900015 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 26931
I20240501 20:01:35.900135 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:35.900372 8865 external_mini_cluster.cc:1428] Setting key 24d915e332f45196989b2c0c520a55e4
I20240501 20:01:35.901525 26998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:35.902340 26998 log.cc:824] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:35.904335 26998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a: No bootstrap required, opened a new log
I20240501 20:01:35.907286 26998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61370bad7b034707b525feb5f516f84a" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45745 } }
I20240501 20:01:35.907514 26998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:35.907584 26998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61370bad7b034707b525feb5f516f84a, State: Initialized, Role: FOLLOWER
I20240501 20:01:35.908063 26998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61370bad7b034707b525feb5f516f84a" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45745 } }
I20240501 20:01:35.908221 26998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:35.908291 26998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:35.908396 26998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:35.908773 27001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:35.901463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.8.168.126:45745)
0501 20:01:35.902171 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:35.902182 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:35.902225 (+ 43us) server_negotiation.cc:407] Connection header received
0501 20:01:35.902932 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:35.902952 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:35.903278 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:35.903646 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:35.904089 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.905257 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:35.905899 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:35.906181 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:35.906560 (+ 379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:35.906582 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:35.906595 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:35.907067 (+ 472us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:35.907122 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:35.907140 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:35.907244 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:35.907340 (+ 96us) server_negotiation.cc:299] Negotiation successful
0501 20:01:35.907569 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":154,"threads_started":1}
I20240501 20:01:35.909324 26998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61370bad7b034707b525feb5f516f84a" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45745 } }
I20240501 20:01:35.909637 26998 leader_election.cc:302] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61370bad7b034707b525feb5f516f84a; no voters:
I20240501 20:01:35.909947 26998 leader_election.cc:288] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:35.910033 27003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:35.910235 27003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [term 1 LEADER]: Becoming Leader. State: Replica: 61370bad7b034707b525feb5f516f84a, State: Running, Role: LEADER
I20240501 20:01:35.910517 27003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61370bad7b034707b525feb5f516f84a" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45745 } }
I20240501 20:01:35.911012 26998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:35.911650 27005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61370bad7b034707b525feb5f516f84a. Latest consensus state: current_term: 1 leader_uuid: "61370bad7b034707b525feb5f516f84a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61370bad7b034707b525feb5f516f84a" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45745 } } }
I20240501 20:01:35.911779 27005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:35.911999 27004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61370bad7b034707b525feb5f516f84a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61370bad7b034707b525feb5f516f84a" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45745 } } }
I20240501 20:01:35.912077 27004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a [sys.catalog]: This master's current role is: LEADER
W20240501 20:01:35.915488 27018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:01:35.915558 27018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:01:35.916051 27012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:35.916750 27012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:35.921478 27012 catalog_manager.cc:1347] Generated new cluster ID: 05557a76938f4c6d83a6e756ed126c0f
I20240501 20:01:35.921545 27012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:35.948161 27012 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:35.949131 27012 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:35.957890 27012 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a: Generated new TSK 0
I20240501 20:01:35.958593 27012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:35.967720 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:45745
--builtin_ntp_servers=127.8.168.84:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240501 20:01:36.072916 27022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:36.073141 27022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:36.073212 27022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:36.076262 27022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:36.076323 27022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:36.076416 27022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 20:01:36.080044 27022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:45745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:36.081012 27022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:36.082109 27022 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:36.089887 27028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:36.090281 27027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:36.090569 27030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:36.091135 27022 server_base.cc:1034] running on GCE node
I20240501 20:01:36.091497 27022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:36.092027 27022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:36.093219 27022 hybrid_clock.cc:648] HybridClock initialized: now 1714593696093198 us; error 32 us; skew 500 ppm
I20240501 20:01:36.095139 27022 webserver.cc:460] Webserver started at http://127.8.168.65:44475/ using document root <none> and password file <none>
I20240501 20:01:36.095672 27022 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:36.095726 27022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:36.095906 27022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:36.097663 27022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "9534da65243a40b9b4598e3160841128"
format_stamp: "Formatted at 2024-05-01 20:01:36 on dist-test-slave-tzct"
server_key: "dae88a4325da46b77eeae902c6529cdf"
server_key_iv: "5847cc937c2dc601e7c66fc8bad4eff5"
server_key_version: "encryptionkey@0"
I20240501 20:01:36.098120 27022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "9534da65243a40b9b4598e3160841128"
format_stamp: "Formatted at 2024-05-01 20:01:36 on dist-test-slave-tzct"
server_key: "dae88a4325da46b77eeae902c6529cdf"
server_key_iv: "5847cc937c2dc601e7c66fc8bad4eff5"
server_key_version: "encryptionkey@0"
I20240501 20:01:36.101341 27022 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:01:36.103482 27036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:36.104485 27022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:36.104594 27022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "9534da65243a40b9b4598e3160841128"
format_stamp: "Formatted at 2024-05-01 20:01:36 on dist-test-slave-tzct"
server_key: "dae88a4325da46b77eeae902c6529cdf"
server_key_iv: "5847cc937c2dc601e7c66fc8bad4eff5"
server_key_version: "encryptionkey@0"
I20240501 20:01:36.104677 27022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:36.129281 27022 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:36.129969 27022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:36.130132 27022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:36.130707 27022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:36.131606 27022 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:36.131655 27022 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:36.131709 27022 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:36.131736 27022 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:36.140054 27022 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:34839
I20240501 20:01:36.140122 27149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:34839 every 8 connection(s)
I20240501 20:01:36.141011 27022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 20:01:36.143494 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 27022
I20240501 20:01:36.143625 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 20:01:36.143983 8865 external_mini_cluster.cc:1428] Setting key f0c2a0690ff06c9d54c0c328ec78b6f5
I20240501 20:01:36.146147 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:45745
--builtin_ntp_servers=127.8.168.84:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240501 20:01:36.149215 27001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.142652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:56223 (local address 127.8.168.126:45745)
0501 20:01:36.142817 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:36.142822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:36.143909 (+ 1087us) server_negotiation.cc:407] Connection header received
0501 20:01:36.144813 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:36.144817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:36.144874 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:36.144960 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:36.146638 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.147140 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.147820 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.147986 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.148670 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:36.148689 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:36.148693 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:36.148707 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:36.148755 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:36.148765 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:36.148853 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:36.148978 (+ 125us) server_negotiation.cc:299] Negotiation successful
0501 20:01:36.149043 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20240501 20:01:36.149855 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.142933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45745 (local address 127.8.168.65:56223)
0501 20:01:36.143684 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:36.143729 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:36.144574 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:36.145137 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:36.145145 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:36.145222 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:36.146484 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:36.146506 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.147274 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.147278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:36.147708 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:36.147716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.147885 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.148443 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 20:01:36.148445 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:36.148479 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 20:01:36.148535 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:36.148854 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:36.148859 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:36.148863 (+ 4us) client_negotiation.cc:769] Sending connection context
0501 20:01:36.148966 (+ 103us) client_negotiation.cc:240] Negotiation successful
0501 20:01:36.149156 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":87,"threads_started":1}
I20240501 20:01:36.150939 27150 heartbeater.cc:347] Connected to a master server at 127.8.168.126:45745
I20240501 20:01:36.151194 27150 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:36.151895 27150 heartbeater.cc:511] Master 127.8.168.126:45745 requested a full tablet report, sending...
I20240501 20:01:36.153508 26962 ts_manager.cc:194] Registered new tserver with Master: 9534da65243a40b9b4598e3160841128 (127.8.168.65:34839)
I20240501 20:01:36.154672 26962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.168.65:56223
W20240501 20:01:36.260802 27153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:36.261034 27153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:36.261093 27153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:36.264183 27153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:36.264250 27153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:36.264339 27153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 20:01:36.268020 27153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:45745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:36.268987 27153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:36.270116 27153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:36.278509 27161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:36.278564 27159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:36.279050 27158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:36.279443 27153 server_base.cc:1034] running on GCE node
I20240501 20:01:36.279844 27153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:36.280443 27153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:36.281603 27153 hybrid_clock.cc:648] HybridClock initialized: now 1714593696281568 us; error 51 us; skew 500 ppm
I20240501 20:01:36.283519 27153 webserver.cc:460] Webserver started at http://127.8.168.66:33177/ using document root <none> and password file <none>
I20240501 20:01:36.284070 27153 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:36.284128 27153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:36.284302 27153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:36.286154 27153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "9fb84d0bf03f46dc9bada27e98f9359a"
format_stamp: "Formatted at 2024-05-01 20:01:36 on dist-test-slave-tzct"
server_key: "0e9927838c4133dccb002574acbf059f"
server_key_iv: "3cfcae83094021aebf3a4cbbd51c98d5"
server_key_version: "encryptionkey@0"
I20240501 20:01:36.286620 27153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "9fb84d0bf03f46dc9bada27e98f9359a"
format_stamp: "Formatted at 2024-05-01 20:01:36 on dist-test-slave-tzct"
server_key: "0e9927838c4133dccb002574acbf059f"
server_key_iv: "3cfcae83094021aebf3a4cbbd51c98d5"
server_key_version: "encryptionkey@0"
I20240501 20:01:36.289994 27153 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:01:36.292311 27167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:36.293401 27153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:36.293516 27153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "9fb84d0bf03f46dc9bada27e98f9359a"
format_stamp: "Formatted at 2024-05-01 20:01:36 on dist-test-slave-tzct"
server_key: "0e9927838c4133dccb002574acbf059f"
server_key_iv: "3cfcae83094021aebf3a4cbbd51c98d5"
server_key_version: "encryptionkey@0"
I20240501 20:01:36.293601 27153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:36.310134 27153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:36.310812 27153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:36.310983 27153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:36.311599 27153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:36.312572 27153 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:36.312620 27153 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:36.312659 27153 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:36.312682 27153 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:36.320623 27153 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:35291
I20240501 20:01:36.320765 27280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:35291 every 8 connection(s)
I20240501 20:01:36.321578 27153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 20:01:36.321992 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 27153
I20240501 20:01:36.322114 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 20:01:36.322414 8865 external_mini_cluster.cc:1428] Setting key 24b30da9a66b19f6e12a0f5e86952fb5
I20240501 20:01:36.324537 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:45745
--builtin_ntp_servers=127.8.168.84:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240501 20:01:36.329982 27001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.323347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:42477 (local address 127.8.168.126:45745)
0501 20:01:36.323496 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:36.323501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:36.324542 (+ 1041us) server_negotiation.cc:407] Connection header received
0501 20:01:36.325412 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:36.325416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:36.325474 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:36.325549 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:36.327059 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.327568 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.328383 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.328512 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.329453 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:36.329472 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:36.329475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:36.329487 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:36.329508 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:36.329515 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:36.329576 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:36.329792 (+ 216us) server_negotiation.cc:299] Negotiation successful
0501 20:01:36.329856 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 20:01:36.330811 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.323711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45745 (local address 127.8.168.66:42477)
0501 20:01:36.324414 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:36.324449 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:36.325213 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:36.325711 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:36.325719 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:36.325798 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:36.326894 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:36.326914 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.327691 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.327696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:36.328271 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:36.328281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.328525 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.329173 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 20:01:36.329176 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:36.329225 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 20:01:36.329300 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:36.329596 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:36.329603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:36.329610 (+ 7us) client_negotiation.cc:769] Sending connection context
0501 20:01:36.329756 (+ 146us) client_negotiation.cc:240] Negotiation successful
0501 20:01:36.330017 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":131,"threads_started":1}
I20240501 20:01:36.331933 27281 heartbeater.cc:347] Connected to a master server at 127.8.168.126:45745
I20240501 20:01:36.332151 27281 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:36.332684 27281 heartbeater.cc:511] Master 127.8.168.126:45745 requested a full tablet report, sending...
I20240501 20:01:36.333781 26962 ts_manager.cc:194] Registered new tserver with Master: 9fb84d0bf03f46dc9bada27e98f9359a (127.8.168.66:35291)
I20240501 20:01:36.334340 26962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.168.66:42477
W20240501 20:01:36.431499 27284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:36.431735 27284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:36.431804 27284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:36.434855 27284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:36.434952 27284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:36.435047 27284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 20:01:36.438613 27284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:45745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:36.439556 27284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:36.440655 27284 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:36.448573 27290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:36.448876 27289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:36.449178 27292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:36.449725 27284 server_base.cc:1034] running on GCE node
I20240501 20:01:36.450116 27284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:36.450690 27284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:36.451833 27284 hybrid_clock.cc:648] HybridClock initialized: now 1714593696451818 us; error 33 us; skew 500 ppm
I20240501 20:01:36.453739 27284 webserver.cc:460] Webserver started at http://127.8.168.67:36845/ using document root <none> and password file <none>
I20240501 20:01:36.454260 27284 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:36.454316 27284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:36.454489 27284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:36.456223 27284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "eda05aeacfb04113b3b374402eac332b"
format_stamp: "Formatted at 2024-05-01 20:01:36 on dist-test-slave-tzct"
server_key: "0234d55626740c6d05e7180f6d6d2d70"
server_key_iv: "0dbd13ee052a0eaf07c7b1e9838615f0"
server_key_version: "encryptionkey@0"
I20240501 20:01:36.456679 27284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "eda05aeacfb04113b3b374402eac332b"
format_stamp: "Formatted at 2024-05-01 20:01:36 on dist-test-slave-tzct"
server_key: "0234d55626740c6d05e7180f6d6d2d70"
server_key_iv: "0dbd13ee052a0eaf07c7b1e9838615f0"
server_key_version: "encryptionkey@0"
I20240501 20:01:36.459949 27284 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 20:01:36.462139 27298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:36.463099 27284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:01:36.463210 27284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "eda05aeacfb04113b3b374402eac332b"
format_stamp: "Formatted at 2024-05-01 20:01:36 on dist-test-slave-tzct"
server_key: "0234d55626740c6d05e7180f6d6d2d70"
server_key_iv: "0dbd13ee052a0eaf07c7b1e9838615f0"
server_key_version: "encryptionkey@0"
I20240501 20:01:36.463295 27284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:36.472009 27284 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:36.472587 27284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:36.472744 27284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:36.473232 27284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:36.474071 27284 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:36.474117 27284 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:36.474153 27284 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:36.474166 27284 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:36.483217 27284 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:40457
I20240501 20:01:36.483240 27411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:40457 every 8 connection(s)
I20240501 20:01:36.484144 27284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 20:01:36.490239 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 27284
I20240501 20:01:36.490374 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 20:01:36.490649 8865 external_mini_cluster.cc:1428] Setting key 281eff7c0c5e26472fcd32254747075a
I20240501 20:01:36.491756 27001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.485838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:35335 (local address 127.8.168.126:45745)
0501 20:01:36.486003 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:36.486007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:36.486849 (+ 842us) server_negotiation.cc:407] Connection header received
0501 20:01:36.487644 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:36.487647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:36.487695 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:36.487768 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:36.489155 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.489653 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.490349 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.490556 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.491289 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:36.491308 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:36.491310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:36.491321 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:36.491341 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:36.491348 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:36.491410 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:36.491576 (+ 166us) server_negotiation.cc:299] Negotiation successful
0501 20:01:36.491624 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20240501 20:01:36.492482 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.486109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:45745 (local address 127.8.168.67:35335)
0501 20:01:36.486721 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:36.486752 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:36.487418 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:36.487956 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:36.487964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:36.488040 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:36.489016 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:36.489031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.489781 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.489784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:36.490216 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:36.490224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.490441 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.491060 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 20:01:36.491063 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:36.491101 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 20:01:36.491156 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:36.491424 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:36.491432 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:36.491438 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 20:01:36.491571 (+ 133us) client_negotiation.cc:240] Negotiation successful
0501 20:01:36.491803 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":103,"threads_started":1}
I20240501 20:01:36.493484 27412 heartbeater.cc:347] Connected to a master server at 127.8.168.126:45745
I20240501 20:01:36.493697 27412 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:36.494221 27412 heartbeater.cc:511] Master 127.8.168.126:45745 requested a full tablet report, sending...
I20240501 20:01:36.495194 26962 ts_manager.cc:194] Registered new tserver with Master: eda05aeacfb04113b3b374402eac332b (127.8.168.67:40457)
I20240501 20:01:36.495764 26962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.168.67:35335
I20240501 20:01:36.504685 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 20:01:36.509953 27001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.506528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44432 (local address 127.8.168.126:45745)
0501 20:01:36.506667 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:36.506671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:36.506770 (+ 99us) server_negotiation.cc:407] Connection header received
0501 20:01:36.506900 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:36.506904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:36.506954 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:36.507019 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:36.507468 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.507961 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.508674 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.508882 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.509465 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:36.509483 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:36.509487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:36.509500 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:36.509521 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:36.509529 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:36.509633 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:36.509762 (+ 129us) server_negotiation.cc:299] Negotiation successful
0501 20:01:36.509796 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20240501 20:01:36.513099 26962 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:01:36.515676 26962 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:01:36.528561 27426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.523891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:40457 (local address 127.0.0.1:47884)
0501 20:01:36.524709 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:36.524773 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:36.524917 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:36.525843 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:36.525849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:36.525953 (+ 104us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:36.526197 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:36.526205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.527196 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.527199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:36.528045 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:36.528058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.528164 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.528226 (+ 62us) client_negotiation.cc:769] Sending connection context
0501 20:01:36.528319 (+ 93us) client_negotiation.cc:240] Negotiation successful
0501 20:01:36.528405 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"spinlock_wait_cycles":164608,"thread_start_us":68,"threads_started":1}
I20240501 20:01:36.528992 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.524265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:34839 (local address 127.0.0.1:46272)
0501 20:01:36.525132 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:36.525144 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:36.525219 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:36.525843 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:36.525849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:36.525886 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:36.526199 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:36.526205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.527622 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.527704 (+ 82us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:36.528563 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:36.528573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.528674 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.528692 (+ 18us) client_negotiation.cc:769] Sending connection context
0501 20:01:36.528780 (+ 88us) client_negotiation.cc:240] Negotiation successful
0501 20:01:36.528838 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"spinlock_wait_cycles":4352,"thread_start_us":84,"threads_started":1}
I20240501 20:01:36.529322 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.523930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:35291 (local address 127.0.0.1:44534)
0501 20:01:36.524709 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:36.524773 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:36.524918 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:36.525845 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:36.525850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:36.525917 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:36.526131 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:36.526138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.527591 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.527595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:36.528522 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:36.528531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.528624 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.528925 (+ 301us) client_negotiation.cc:769] Sending connection context
0501 20:01:36.528983 (+ 58us) client_negotiation.cc:240] Negotiation successful
0501 20:01:36.529041 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"spinlock_wait_cycles":12288,"thread_start_us":117,"threads_started":1}
I20240501 20:01:36.529721 27427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.524611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46272 (local address 127.8.168.65:34839)
0501 20:01:36.525080 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:36.525087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:36.525305 (+ 218us) server_negotiation.cc:407] Connection header received
0501 20:01:36.525387 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:36.525391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:36.525527 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:36.525675 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:36.526306 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.527479 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.528667 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.529229 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.529402 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:36.529486 (+ 84us) server_negotiation.cc:299] Negotiation successful
0501 20:01:36.529585 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":108,"threads_started":1}
I20240501 20:01:36.529897 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.524047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47884 (local address 127.8.168.67:40457)
0501 20:01:36.525201 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:36.525207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:36.525232 (+ 25us) server_negotiation.cc:407] Connection header received
0501 20:01:36.525323 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:36.525329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:36.525527 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:36.525675 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:36.526334 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.527078 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.528901 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.529470 (+ 569us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.529622 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:36.529691 (+ 69us) server_negotiation.cc:299] Negotiation successful
0501 20:01:36.529763 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1054,"thread_start_us":161,"threads_started":1}
I20240501 20:01:36.530251 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.524675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.8.168.66:35291)
0501 20:01:36.525342 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:36.525347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:36.525370 (+ 23us) server_negotiation.cc:407] Connection header received
0501 20:01:36.525457 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:36.525462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:36.525638 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:36.525821 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:36.526252 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.527471 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.529307 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.529834 (+ 527us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.529979 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:36.530042 (+ 63us) server_negotiation.cc:299] Negotiation successful
0501 20:01:36.530127 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":116,"threads_started":1}
I20240501 20:01:36.532377 27346 tablet_service.cc:1467] Processing CreateTablet for tablet ce6fe5a090c14662899d2117c11473f2 (DEFAULT_TABLE table=test-table [id=2ac038fde88a4bcc8596e88af2d904b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:36.532385 27215 tablet_service.cc:1467] Processing CreateTablet for tablet ce6fe5a090c14662899d2117c11473f2 (DEFAULT_TABLE table=test-table [id=2ac038fde88a4bcc8596e88af2d904b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:36.532388 27084 tablet_service.cc:1467] Processing CreateTablet for tablet ce6fe5a090c14662899d2117c11473f2 (DEFAULT_TABLE table=test-table [id=2ac038fde88a4bcc8596e88af2d904b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:36.533277 27346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6fe5a090c14662899d2117c11473f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:36.533285 27084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6fe5a090c14662899d2117c11473f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:36.533285 27215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6fe5a090c14662899d2117c11473f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:36.538884 27429 tablet_bootstrap.cc:492] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b: Bootstrap starting.
I20240501 20:01:36.540138 27430 tablet_bootstrap.cc:492] T ce6fe5a090c14662899d2117c11473f2 P 9534da65243a40b9b4598e3160841128: Bootstrap starting.
I20240501 20:01:36.541044 27431 tablet_bootstrap.cc:492] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a: Bootstrap starting.
I20240501 20:01:36.541232 27429 tablet_bootstrap.cc:654] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:36.541937 27430 tablet_bootstrap.cc:654] T ce6fe5a090c14662899d2117c11473f2 P 9534da65243a40b9b4598e3160841128: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:36.542057 27429 log.cc:824] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:36.542608 27430 log.cc:824] T ce6fe5a090c14662899d2117c11473f2 P 9534da65243a40b9b4598e3160841128: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:36.542834 27431 tablet_bootstrap.cc:654] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:36.543514 27431 log.cc:824] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:36.544096 27429 tablet_bootstrap.cc:492] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b: No bootstrap required, opened a new log
I20240501 20:01:36.544179 27430 tablet_bootstrap.cc:492] T ce6fe5a090c14662899d2117c11473f2 P 9534da65243a40b9b4598e3160841128: No bootstrap required, opened a new log
I20240501 20:01:36.544267 27429 ts_tablet_manager.cc:1367] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:01:36.544342 27430 ts_tablet_manager.cc:1367] T ce6fe5a090c14662899d2117c11473f2 P 9534da65243a40b9b4598e3160841128: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:01:36.545274 27431 tablet_bootstrap.cc:492] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a: No bootstrap required, opened a new log
I20240501 20:01:36.545442 27431 ts_tablet_manager.cc:1367] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 20:01:36.547662 27430 raft_consensus.cc:348] T ce6fe5a090c14662899d2117c11473f2 P 9534da65243a40b9b4598e3160841128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda05aeacfb04113b3b374402eac332b" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40457 } } peers { permanent_uuid: "9fb84d0bf03f46dc9bada27e98f9359a" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35291 } } peers { permanent_uuid: "9534da65243a40b9b4598e3160841128" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34839 } }
I20240501 20:01:36.547859 27430 raft_consensus.cc:374] T ce6fe5a090c14662899d2117c11473f2 P 9534da65243a40b9b4598e3160841128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:36.547921 27430 raft_consensus.cc:729] T ce6fe5a090c14662899d2117c11473f2 P 9534da65243a40b9b4598e3160841128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9534da65243a40b9b4598e3160841128, State: Initialized, Role: FOLLOWER
I20240501 20:01:36.548177 27429 raft_consensus.cc:348] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda05aeacfb04113b3b374402eac332b" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40457 } } peers { permanent_uuid: "9fb84d0bf03f46dc9bada27e98f9359a" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35291 } } peers { permanent_uuid: "9534da65243a40b9b4598e3160841128" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34839 } }
I20240501 20:01:36.548380 27429 raft_consensus.cc:374] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:36.548324 27430 consensus_queue.cc:260] T ce6fe5a090c14662899d2117c11473f2 P 9534da65243a40b9b4598e3160841128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda05aeacfb04113b3b374402eac332b" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40457 } } peers { permanent_uuid: "9fb84d0bf03f46dc9bada27e98f9359a" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35291 } } peers { permanent_uuid: "9534da65243a40b9b4598e3160841128" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34839 } }
I20240501 20:01:36.548504 27429 raft_consensus.cc:729] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eda05aeacfb04113b3b374402eac332b, State: Initialized, Role: FOLLOWER
I20240501 20:01:36.548971 27429 consensus_queue.cc:260] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda05aeacfb04113b3b374402eac332b" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40457 } } peers { permanent_uuid: "9fb84d0bf03f46dc9bada27e98f9359a" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35291 } } peers { permanent_uuid: "9534da65243a40b9b4598e3160841128" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34839 } }
I20240501 20:01:36.549162 27430 ts_tablet_manager.cc:1398] T ce6fe5a090c14662899d2117c11473f2 P 9534da65243a40b9b4598e3160841128: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 20:01:36.549082 27431 raft_consensus.cc:348] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda05aeacfb04113b3b374402eac332b" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40457 } } peers { permanent_uuid: "9fb84d0bf03f46dc9bada27e98f9359a" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35291 } } peers { permanent_uuid: "9534da65243a40b9b4598e3160841128" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34839 } }
I20240501 20:01:36.549278 27431 raft_consensus.cc:374] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:36.549341 27431 raft_consensus.cc:729] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb84d0bf03f46dc9bada27e98f9359a, State: Initialized, Role: FOLLOWER
I20240501 20:01:36.549539 27150 heartbeater.cc:503] Master 127.8.168.126:45745 was elected leader, sending a full tablet report...
I20240501 20:01:36.549721 27431 consensus_queue.cc:260] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda05aeacfb04113b3b374402eac332b" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40457 } } peers { permanent_uuid: "9fb84d0bf03f46dc9bada27e98f9359a" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35291 } } peers { permanent_uuid: "9534da65243a40b9b4598e3160841128" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34839 } }
I20240501 20:01:36.549839 27429 ts_tablet_manager.cc:1398] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240501 20:01:36.550158 27412 heartbeater.cc:503] Master 127.8.168.126:45745 was elected leader, sending a full tablet report...
I20240501 20:01:36.550369 27281 heartbeater.cc:503] Master 127.8.168.126:45745 was elected leader, sending a full tablet report...
I20240501 20:01:36.550724 27431 ts_tablet_manager.cc:1398] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
W20240501 20:01:36.572548 27282 tablet.cc:2369] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:01:36.642119 27151 tablet.cc:2369] T ce6fe5a090c14662899d2117c11473f2 P 9534da65243a40b9b4598e3160841128: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:01:36.719069 27437 raft_consensus.cc:482] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.719257 27437 raft_consensus.cc:504] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda05aeacfb04113b3b374402eac332b" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40457 } } peers { permanent_uuid: "9fb84d0bf03f46dc9bada27e98f9359a" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35291 } } peers { permanent_uuid: "9534da65243a40b9b4598e3160841128" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34839 } }
I20240501 20:01:36.720310 27437 leader_election.cc:288] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eda05aeacfb04113b3b374402eac332b (127.8.168.67:40457), 9534da65243a40b9b4598e3160841128 (127.8.168.65:34839)
I20240501 20:01:36.723737 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.720542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:40457 (local address 127.8.168.66:50727)
0501 20:01:36.720768 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:36.720782 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:36.720889 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:36.721146 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:36.721148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:36.721163 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:36.721415 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:36.721422 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.722378 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.722381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:36.723317 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:36.723328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.723444 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.723468 (+ 24us) client_negotiation.cc:769] Sending connection context
0501 20:01:36.723519 (+ 51us) client_negotiation.cc:240] Negotiation successful
0501 20:01:36.723584 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20240501 20:01:36.723748 27438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.720561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:34839 (local address 127.8.168.66:41601)
0501 20:01:36.720924 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:36.720937 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:36.721013 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:36.721243 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:36.721246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:36.721259 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:36.721512 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:36.721520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.722507 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.722510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:36.723368 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:36.723376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.723472 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.723484 (+ 12us) client_negotiation.cc:769] Sending connection context
0501 20:01:36.723528 (+ 44us) client_negotiation.cc:240] Negotiation successful
0501 20:01:36.723581 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":164,"threads_started":1}
I20240501 20:01:36.724153 27427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.720697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:41601 (local address 127.8.168.65:34839)
0501 20:01:36.720840 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:36.720845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:36.721010 (+ 165us) server_negotiation.cc:407] Connection header received
0501 20:01:36.721096 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:36.721099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:36.721151 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:36.721226 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:36.721699 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.722372 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.723459 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.723904 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.723940 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:36.723989 (+ 49us) server_negotiation.cc:299] Negotiation successful
0501 20:01:36.724039 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20240501 20:01:36.724153 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.720639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:50727 (local address 127.8.168.67:40457)
0501 20:01:36.720775 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:36.720780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:36.720870 (+ 90us) server_negotiation.cc:407] Connection header received
0501 20:01:36.720977 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:36.720980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:36.721040 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:36.721127 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:36.721528 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.722255 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.723440 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.723911 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.723940 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:36.723989 (+ 49us) server_negotiation.cc:299] Negotiation successful
0501 20:01:36.724039 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240501 20:01:36.724751 27366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6fe5a090c14662899d2117c11473f2" candidate_uuid: "9fb84d0bf03f46dc9bada27e98f9359a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eda05aeacfb04113b3b374402eac332b" is_pre_election: true
I20240501 20:01:36.724920 27104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6fe5a090c14662899d2117c11473f2" candidate_uuid: "9fb84d0bf03f46dc9bada27e98f9359a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9534da65243a40b9b4598e3160841128" is_pre_election: true
I20240501 20:01:36.725044 27366 raft_consensus.cc:2454] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fb84d0bf03f46dc9bada27e98f9359a in term 0.
I20240501 20:01:36.725239 27104 raft_consensus.cc:2454] T ce6fe5a090c14662899d2117c11473f2 P 9534da65243a40b9b4598e3160841128 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fb84d0bf03f46dc9bada27e98f9359a in term 0.
I20240501 20:01:36.725517 27169 leader_election.cc:302] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fb84d0bf03f46dc9bada27e98f9359a, eda05aeacfb04113b3b374402eac332b; no voters:
I20240501 20:01:36.725739 27437 raft_consensus.cc:2781] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:36.725809 27437 raft_consensus.cc:482] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:36.725845 27437 raft_consensus.cc:3037] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:36.726716 27437 raft_consensus.cc:504] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda05aeacfb04113b3b374402eac332b" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40457 } } peers { permanent_uuid: "9fb84d0bf03f46dc9bada27e98f9359a" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35291 } } peers { permanent_uuid: "9534da65243a40b9b4598e3160841128" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34839 } }
I20240501 20:01:36.727041 27437 leader_election.cc:288] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [CANDIDATE]: Term 1 election: Requested vote from peers eda05aeacfb04113b3b374402eac332b (127.8.168.67:40457), 9534da65243a40b9b4598e3160841128 (127.8.168.65:34839)
I20240501 20:01:36.727347 27366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6fe5a090c14662899d2117c11473f2" candidate_uuid: "9fb84d0bf03f46dc9bada27e98f9359a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eda05aeacfb04113b3b374402eac332b"
I20240501 20:01:36.727409 27104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6fe5a090c14662899d2117c11473f2" candidate_uuid: "9fb84d0bf03f46dc9bada27e98f9359a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9534da65243a40b9b4598e3160841128"
I20240501 20:01:36.727459 27366 raft_consensus.cc:3037] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:36.727526 27104 raft_consensus.cc:3037] T ce6fe5a090c14662899d2117c11473f2 P 9534da65243a40b9b4598e3160841128 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:36.728354 27366 raft_consensus.cc:2454] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fb84d0bf03f46dc9bada27e98f9359a in term 1.
I20240501 20:01:36.728490 27104 raft_consensus.cc:2454] T ce6fe5a090c14662899d2117c11473f2 P 9534da65243a40b9b4598e3160841128 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fb84d0bf03f46dc9bada27e98f9359a in term 1.
I20240501 20:01:36.728627 27169 leader_election.cc:302] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fb84d0bf03f46dc9bada27e98f9359a, eda05aeacfb04113b3b374402eac332b; no voters:
I20240501 20:01:36.728816 27437 raft_consensus.cc:2781] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:36.729014 27437 raft_consensus.cc:686] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [term 1 LEADER]: Becoming Leader. State: Replica: 9fb84d0bf03f46dc9bada27e98f9359a, State: Running, Role: LEADER
I20240501 20:01:36.729305 27437 consensus_queue.cc:237] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda05aeacfb04113b3b374402eac332b" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40457 } } peers { permanent_uuid: "9fb84d0bf03f46dc9bada27e98f9359a" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35291 } } peers { permanent_uuid: "9534da65243a40b9b4598e3160841128" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34839 } }
I20240501 20:01:36.732336 26959 catalog_manager.cc:5484] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fb84d0bf03f46dc9bada27e98f9359a (127.8.168.66). New cstate: current_term: 1 leader_uuid: "9fb84d0bf03f46dc9bada27e98f9359a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda05aeacfb04113b3b374402eac332b" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fb84d0bf03f46dc9bada27e98f9359a" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9534da65243a40b9b4598e3160841128" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34839 } health_report { overall_health: UNKNOWN } } }
W20240501 20:01:36.734958 27413 tablet.cc:2369] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:01:36.765761 27428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.762441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.8.168.66:35291)
0501 20:01:36.762570 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:36.762574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:36.762588 (+ 14us) server_negotiation.cc:407] Connection header received
0501 20:01:36.762714 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:36.762717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:36.762769 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:36.762868 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:01:36.763225 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.763801 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:36.764452 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:36.764642 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:36.764936 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:01:36.765342 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:01:36.765496 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:36.765606 (+ 110us) server_negotiation.cc:299] Negotiation successful
0501 20:01:36.765640 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240501 20:01:36.771466 27366 raft_consensus.cc:1261] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b [term 1 FOLLOWER]: Refusing update from remote peer 9fb84d0bf03f46dc9bada27e98f9359a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:36.771476 27104 raft_consensus.cc:1261] T ce6fe5a090c14662899d2117c11473f2 P 9534da65243a40b9b4598e3160841128 [term 1 FOLLOWER]: Refusing update from remote peer 9fb84d0bf03f46dc9bada27e98f9359a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:36.772147 27439 consensus_queue.cc:1035] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eda05aeacfb04113b3b374402eac332b" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 40457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:36.772352 27437 consensus_queue.cc:1035] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9534da65243a40b9b4598e3160841128" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 34839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:36.779810 27443 mvcc.cc:205] Tried to move back new op lower bound from 7022975781968101376 to 7022975781805514752. Current Snapshot: MvccSnapshot[applied={T|T < 7022975781968101376}]
I20240501 20:01:36.789294 26959 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44432:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:01:36.789456 26959 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44432:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:01:36.791757 26959 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 61370bad7b034707b525feb5f516f84a: Sending DeleteTablet for 3 replicas of tablet ce6fe5a090c14662899d2117c11473f2
I20240501 20:01:36.792503 27215 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6fe5a090c14662899d2117c11473f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:36 UTC) from {username='slave'} at 127.0.0.1:44534
I20240501 20:01:36.792503 27084 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6fe5a090c14662899d2117c11473f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:36 UTC) from {username='slave'} at 127.0.0.1:46272
I20240501 20:01:36.793020 27452 tablet_replica.cc:330] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a: stopping tablet replica
I20240501 20:01:36.792994 27346 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6fe5a090c14662899d2117c11473f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:36 UTC) from {username='slave'} at 127.0.0.1:47884
I20240501 20:01:36.793201 27453 tablet_replica.cc:330] T ce6fe5a090c14662899d2117c11473f2 P 9534da65243a40b9b4598e3160841128: stopping tablet replica
I20240501 20:01:36.793284 27452 raft_consensus.cc:2229] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:36.793540 27453 raft_consensus.cc:2229] T ce6fe5a090c14662899d2117c11473f2 P 9534da65243a40b9b4598e3160841128 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:36.793690 27452 raft_consensus.cc:2258] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:36.794153 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 27022
I20240501 20:01:36.795734 27454 tablet_replica.cc:330] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b: stopping tablet replica
I20240501 20:01:36.796036 27454 raft_consensus.cc:2229] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:36.796258 27452 ts_tablet_manager.cc:1875] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:01:36.796456 27454 raft_consensus.cc:2258] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:36.797927 27454 ts_tablet_manager.cc:1875] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:01:36.798935 27452 ts_tablet_manager.cc:1888] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:01:36.799028 27452 log.cc:1196] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/ce6fe5a090c14662899d2117c11473f2
I20240501 20:01:36.799320 27452 ts_tablet_manager.cc:1909] T ce6fe5a090c14662899d2117c11473f2 P 9fb84d0bf03f46dc9bada27e98f9359a: Deleting consensus metadata
I20240501 20:01:36.800765 27454 ts_tablet_manager.cc:1888] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:01:36.800926 27454 log.cc:1196] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/ce6fe5a090c14662899d2117c11473f2
I20240501 20:01:36.801230 26946 catalog_manager.cc:4830] TS 9fb84d0bf03f46dc9bada27e98f9359a (127.8.168.66:35291): tablet ce6fe5a090c14662899d2117c11473f2 (table test-table [id=2ac038fde88a4bcc8596e88af2d904b2]) successfully deleted
I20240501 20:01:36.801290 27454 ts_tablet_manager.cc:1909] T ce6fe5a090c14662899d2117c11473f2 P eda05aeacfb04113b3b374402eac332b: Deleting consensus metadata
W20240501 20:01:36.801587 26948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.168.65:34839 (error 108)
I20240501 20:01:36.802476 26947 catalog_manager.cc:4830] TS eda05aeacfb04113b3b374402eac332b (127.8.168.67:40457): tablet ce6fe5a090c14662899d2117c11473f2 (table test-table [id=2ac038fde88a4bcc8596e88af2d904b2]) successfully deleted
I20240501 20:01:36.802512 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:36.802239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:34839 (local address 127.0.0.1:46288)
0501 20:01:36.802360 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:36.802426 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:34839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20240501 20:01:36.802599 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 27153
W20240501 20:01:36.803925 26948 catalog_manager.cc:4573] TS 9534da65243a40b9b4598e3160841128 (127.8.168.65:34839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce6fe5a090c14662899d2117c11473f2: Network error: Client connection negotiation failed: client connection to 127.8.168.65:34839: connect: Connection refused (error 111)
I20240501 20:01:36.807420 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 27284
I20240501 20:01:36.811836 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 26931
2024-05-01T20:01:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1113 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:36 dist-test-slave-tzct krb5kdc[27458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:36 dist-test-slave-tzct krb5kdc[27458](info): set up 2 sockets
May 01 20:01:36 dist-test-slave-tzct krb5kdc[27458](info): commencing operation
krb5kdc: starting...
W20240501 20:01:37.700711 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.870s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:37 dist-test-slave-tzct krb5kdc[27458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 20:01:37.811801 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:34211
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:34211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:37.917575 27471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:37.917802 27471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:37.917851 27471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:37.920799 27471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:37.920845 27471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:37.920866 27471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:37.920881 27471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:37.920902 27471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:37.925021 27471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:34211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:34211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:37.925814 27471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:37.926601 27471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:37.933377 27476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:37.933790 27479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:37.934185 27477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:37.934736 27471 server_base.cc:1034] running on GCE node
I20240501 20:01:37.935206 27471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:37.936062 27471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:37.937198 27471 hybrid_clock.cc:648] HybridClock initialized: now 1714593697937181 us; error 35 us; skew 500 ppm
May 01 20:01:37 dist-test-slave-tzct krb5kdc[27458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593697, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:37.939600 27471 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:01:37.940474 27471 webserver.cc:460] Webserver started at http://127.8.168.126:37883/ using document root <none> and password file <none>
I20240501 20:01:37.941008 27471 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:37.941066 27471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:37.941226 27471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:37.942873 27471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "280d641558804180805d8b524f1a8746"
format_stamp: "Formatted at 2024-05-01 20:01:37 on dist-test-slave-tzct"
server_key: "6dd58036d6f6c9eae9d5cfb38d086d6f"
server_key_iv: "9e73d935df059869548db5471ac564be"
server_key_version: "encryptionkey@0"
I20240501 20:01:37.943327 27471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "280d641558804180805d8b524f1a8746"
format_stamp: "Formatted at 2024-05-01 20:01:37 on dist-test-slave-tzct"
server_key: "6dd58036d6f6c9eae9d5cfb38d086d6f"
server_key_iv: "9e73d935df059869548db5471ac564be"
server_key_version: "encryptionkey@0"
I20240501 20:01:37.946452 27471 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240501 20:01:37.948458 27486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:37.949496 27471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:01:37.949609 27471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "280d641558804180805d8b524f1a8746"
format_stamp: "Formatted at 2024-05-01 20:01:37 on dist-test-slave-tzct"
server_key: "6dd58036d6f6c9eae9d5cfb38d086d6f"
server_key_iv: "9e73d935df059869548db5471ac564be"
server_key_version: "encryptionkey@0"
I20240501 20:01:37.949693 27471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:37.989789 27471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:37.992350 27471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:37.992516 27471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:37.998867 27471 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:34211
I20240501 20:01:37.998878 27538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:34211 every 8 connection(s)
I20240501 20:01:37.999819 27471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:38.002542 27539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:38.006956 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 27471
I20240501 20:01:38.007082 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:38.007352 8865 external_mini_cluster.cc:1428] Setting key 47ffaa1cfcdce3c0c3ffe599a7224745
I20240501 20:01:38.007380 27539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746: Bootstrap starting.
I20240501 20:01:38.009573 27539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:38.010267 27539 log.cc:824] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:38.011986 27539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746: No bootstrap required, opened a new log
May 01 20:01:38 dist-test-slave-tzct krb5kdc[27458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:01:38.015517 27539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280d641558804180805d8b524f1a8746" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 34211 } }
I20240501 20:01:38.015698 27539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:38.015733 27539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 280d641558804180805d8b524f1a8746, State: Initialized, Role: FOLLOWER
I20240501 20:01:38.016150 27539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280d641558804180805d8b524f1a8746" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 34211 } }
I20240501 20:01:38.016263 27539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:38.016340 27539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:38.016458 27539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:38.017246 27539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280d641558804180805d8b524f1a8746" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 34211 } }
I20240501 20:01:38.017554 27539 leader_election.cc:302] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 280d641558804180805d8b524f1a8746; no voters:
I20240501 20:01:38.017848 27539 leader_election.cc:288] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:38.017928 27544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:38.018147 27544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [term 1 LEADER]: Becoming Leader. State: Replica: 280d641558804180805d8b524f1a8746, State: Running, Role: LEADER
I20240501 20:01:38.018508 27544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280d641558804180805d8b524f1a8746" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 34211 } }
I20240501 20:01:38.019009 27539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:38.019637 27544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "280d641558804180805d8b524f1a8746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280d641558804180805d8b524f1a8746" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 34211 } } }
I20240501 20:01:38.019845 27544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:38.019970 27545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 280d641558804180805d8b524f1a8746. Latest consensus state: current_term: 1 leader_uuid: "280d641558804180805d8b524f1a8746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280d641558804180805d8b524f1a8746" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 34211 } } }
I20240501 20:01:38.020102 27545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:38.020241 27553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:38.020677 27542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:38.008579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.8.168.126:34211)
0501 20:01:38.008978 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:38.008988 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:38.009023 (+ 35us) server_negotiation.cc:407] Connection header received
0501 20:01:38.009617 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:38.009638 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:38.009935 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:38.010230 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:38.011157 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.011986 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:38.012816 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.013030 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:38.015442 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:38.015460 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:38.015471 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:38.015536 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:38.017514 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:38.017966 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:38.017972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:38.017977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:38.018098 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:38.018377 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:38.018380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:38.018382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:38.018811 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:38.019020 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:38.019588 (+ 568us) server_negotiation.cc:299] Negotiation successful
0501 20:01:38.019850 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":126,"threads_started":1}
I20240501 20:01:38.022876 27553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:38.027640 27553 catalog_manager.cc:1347] Generated new cluster ID: 0a4062ae03aa44d099384a9391955852
I20240501 20:01:38.027707 27553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:38.044711 27553 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:38.045966 27553 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:38.060561 27553 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746: Generated new TSK 0
I20240501 20:01:38.061424 27553 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.168.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
Entry for principal kudu/127.8.168.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab.
I20240501 20:01:38.088450 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:34211
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:38.190737 27565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:38.190949 27565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:38.191004 27565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:38.193956 27565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:38.194017 27565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:38.194089 27565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 20:01:38.198091 27565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.65.keytab
--principal=kudu/127.8.168.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:34211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:38.198907 27565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:38.199684 27565 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:38.207216 27571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:38.207657 27573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:38.208014 27570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:38.208616 27565 server_base.cc:1034] running on GCE node
I20240501 20:01:38.208981 27565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:38.209488 27565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:38.210659 27565 hybrid_clock.cc:648] HybridClock initialized: now 1714593698210645 us; error 25 us; skew 500 ppm
May 01 20:01:38 dist-test-slave-tzct krb5kdc[27458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593698, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:38.213172 27565 init.cc:377] Logged in from keytab as kudu/127.8.168.65@KRBTEST.COM (short username kudu)
I20240501 20:01:38.214025 27565 webserver.cc:460] Webserver started at http://127.8.168.65:33599/ using document root <none> and password file <none>
I20240501 20:01:38.214522 27565 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:38.214596 27565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:38.214758 27565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:38.216477 27565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "fdd3e9d4f4174f05b85cfb1ac8f0135b"
format_stamp: "Formatted at 2024-05-01 20:01:38 on dist-test-slave-tzct"
server_key: "531b77ae65ca9595a0bfebacab6e2278"
server_key_iv: "6d4368fe691b6818521379e6bf945d4e"
server_key_version: "encryptionkey@0"
I20240501 20:01:38.216941 27565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "fdd3e9d4f4174f05b85cfb1ac8f0135b"
format_stamp: "Formatted at 2024-05-01 20:01:38 on dist-test-slave-tzct"
server_key: "531b77ae65ca9595a0bfebacab6e2278"
server_key_iv: "6d4368fe691b6818521379e6bf945d4e"
server_key_version: "encryptionkey@0"
I20240501 20:01:38.220019 27565 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 20:01:38.222108 27580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:38.223037 27565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:01:38.223130 27565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "fdd3e9d4f4174f05b85cfb1ac8f0135b"
format_stamp: "Formatted at 2024-05-01 20:01:38 on dist-test-slave-tzct"
server_key: "531b77ae65ca9595a0bfebacab6e2278"
server_key_iv: "6d4368fe691b6818521379e6bf945d4e"
server_key_version: "encryptionkey@0"
I20240501 20:01:38.223207 27565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:38.242165 27565 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:38.244808 27565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:38.244961 27565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:38.245474 27565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:38.246233 27565 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:38.246279 27565 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:38.246330 27565 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:38.246357 27565 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:38.254626 27565 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:39369
I20240501 20:01:38.254657 27693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:39369 every 8 connection(s)
I20240501 20:01:38.255481 27565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
May 01 20:01:38 dist-test-slave-tzct krb5kdc[27458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593698, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.65@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:01:38.263672 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 27565
I20240501 20:01:38.263789 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 20:01:38.264060 8865 external_mini_cluster.cc:1428] Setting key 79315d844fe0bfbf8a95c18681440852
I20240501 20:01:38.266978 27542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:38.257083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:54813 (local address 127.8.168.126:34211)
0501 20:01:38.257209 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:38.257213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:38.257855 (+ 642us) server_negotiation.cc:407] Connection header received
0501 20:01:38.258631 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:38.258635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:38.258691 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:38.258762 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:38.260326 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.260886 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:38.261506 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.261684 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:38.263758 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:38.263779 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:38.263782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:38.263812 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:38.265242 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:38.265687 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:38.265690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:38.265692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:38.265734 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:38.265991 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:38.265994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:38.265996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:38.266137 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:38.266219 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:38.266686 (+ 467us) server_negotiation.cc:299] Negotiation successful
0501 20:01:38.266817 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240501 20:01:38.267658 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:38.257337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:34211 (local address 127.8.168.65:54813)
0501 20:01:38.257719 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:38.257753 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:38.258434 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:38.258950 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:38.258958 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:38.259302 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:38.260183 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:38.260196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.261002 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:38.261005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:38.261394 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:38.261402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.261600 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:38.262224 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:01:38.262249 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:38.263607 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:38.265355 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:38.265360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:38.265371 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:38.265597 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:38.265810 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:38.265813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:38.265815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:38.265904 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:38.266233 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:38.266239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:38.266463 (+ 224us) client_negotiation.cc:769] Sending connection context
0501 20:01:38.266634 (+ 171us) client_negotiation.cc:240] Negotiation successful
0501 20:01:38.266905 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":96,"threads_started":1}
I20240501 20:01:38.268760 27694 heartbeater.cc:347] Connected to a master server at 127.8.168.126:34211
I20240501 20:01:38.268975 27694 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:38.269434 27694 heartbeater.cc:511] Master 127.8.168.126:34211 requested a full tablet report, sending...
I20240501 20:01:38.270794 27503 ts_manager.cc:194] Registered new tserver with Master: fdd3e9d4f4174f05b85cfb1ac8f0135b (127.8.168.65:39369)
WARNING: no policy specified for kudu/127.8.168.66@KRBTEST.COM; defaulting to no policy
I20240501 20:01:38.271929 27503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.65@KRBTEST.COM'} at 127.8.168.65:54813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
Entry for principal kudu/127.8.168.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab.
I20240501 20:01:38.288370 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:34211
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:38.390403 27699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:38.390623 27699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:38.390700 27699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:38.393757 27699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:38.393832 27699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:38.393930 27699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 20:01:38.398020 27699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.66.keytab
--principal=kudu/127.8.168.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:34211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:38.398883 27699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:38.399684 27699 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:38.407444 27704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:38.407864 27707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:38.408041 27705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:38.408485 27699 server_base.cc:1034] running on GCE node
I20240501 20:01:38.408880 27699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:38.409416 27699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:38.410585 27699 hybrid_clock.cc:648] HybridClock initialized: now 1714593698410571 us; error 31 us; skew 500 ppm
May 01 20:01:38 dist-test-slave-tzct krb5kdc[27458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593698, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:38.413269 27699 init.cc:377] Logged in from keytab as kudu/127.8.168.66@KRBTEST.COM (short username kudu)
I20240501 20:01:38.414182 27699 webserver.cc:460] Webserver started at http://127.8.168.66:40453/ using document root <none> and password file <none>
I20240501 20:01:38.414716 27699 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:38.414770 27699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:38.414924 27699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:38.416590 27699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "f10d9e66d056437db3a86bcc8b98436c"
format_stamp: "Formatted at 2024-05-01 20:01:38 on dist-test-slave-tzct"
server_key: "a8f0d4be639d2123cacf96e6fe221f57"
server_key_iv: "20a327f24c4553c246953c3736ac5389"
server_key_version: "encryptionkey@0"
I20240501 20:01:38.417048 27699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "f10d9e66d056437db3a86bcc8b98436c"
format_stamp: "Formatted at 2024-05-01 20:01:38 on dist-test-slave-tzct"
server_key: "a8f0d4be639d2123cacf96e6fe221f57"
server_key_iv: "20a327f24c4553c246953c3736ac5389"
server_key_version: "encryptionkey@0"
I20240501 20:01:38.420193 27699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:01:38.422435 27714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:38.423491 27699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240501 20:01:38.423597 27699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "f10d9e66d056437db3a86bcc8b98436c"
format_stamp: "Formatted at 2024-05-01 20:01:38 on dist-test-slave-tzct"
server_key: "a8f0d4be639d2123cacf96e6fe221f57"
server_key_iv: "20a327f24c4553c246953c3736ac5389"
server_key_version: "encryptionkey@0"
I20240501 20:01:38.423688 27699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:38.445910 27699 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:38.448812 27699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:38.448976 27699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:38.449505 27699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:38.450381 27699 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:38.450428 27699 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:38.450471 27699 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:38.450500 27699 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:38.460176 27699 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:40681
I20240501 20:01:38.460194 27827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:40681 every 8 connection(s)
I20240501 20:01:38.461158 27699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 20:01:38.463734 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 27699
I20240501 20:01:38.463866 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 20:01:38.464160 8865 external_mini_cluster.cc:1428] Setting key 82dafe9449b70b09e0e5bcccd408357d
May 01 20:01:38 dist-test-slave-tzct krb5kdc[27458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593698, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.66@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.168.67@KRBTEST.COM; defaulting to no policy
I20240501 20:01:38.473847 27542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:38.462888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:36089 (local address 127.8.168.126:34211)
0501 20:01:38.463048 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:38.463052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:38.463760 (+ 708us) server_negotiation.cc:407] Connection header received
0501 20:01:38.464764 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:38.464768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:38.464817 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:38.464886 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:38.466848 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.467347 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:38.467994 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.468174 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:38.470491 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:38.470511 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:38.470513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:38.470540 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:38.471889 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:38.472506 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:38.472509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:38.472510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:38.472557 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:38.472860 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:38.472862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:38.472863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:38.473014 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:38.473090 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:38.473613 (+ 523us) server_negotiation.cc:299] Negotiation successful
0501 20:01:38.473719 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20240501 20:01:38.474633 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:38.463115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:34211 (local address 127.8.168.66:36089)
0501 20:01:38.463593 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:38.463639 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:38.464513 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:38.465069 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:38.465077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:38.465522 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:38.466698 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:38.466715 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.467470 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:38.467473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:38.467861 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:38.467869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.468094 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:38.468691 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:01:38.468730 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:38.470320 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:38.472012 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:38.472019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:38.472031 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:38.472361 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:38.472646 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:38.472649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:38.472651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:38.472781 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:38.473106 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:38.473111 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:38.473398 (+ 287us) client_negotiation.cc:769] Sending connection context
0501 20:01:38.473582 (+ 184us) client_negotiation.cc:240] Negotiation successful
0501 20:01:38.473882 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.67@KRBTEST.COM" created.
I20240501 20:01:38.475807 27828 heartbeater.cc:347] Connected to a master server at 127.8.168.126:34211
I20240501 20:01:38.476044 27828 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:38.476645 27828 heartbeater.cc:511] Master 127.8.168.126:34211 requested a full tablet report, sending...
I20240501 20:01:38.477734 27503 ts_manager.cc:194] Registered new tserver with Master: f10d9e66d056437db3a86bcc8b98436c (127.8.168.66:40681)
I20240501 20:01:38.478286 27503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.66@KRBTEST.COM'} at 127.8.168.66:36089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
Entry for principal kudu/127.8.168.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab.
I20240501 20:01:38.491129 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:34211
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:38.598299 27833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:38.598505 27833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:38.598563 27833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:38.601550 27833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:38.601603 27833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:38.601701 27833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 20:01:38.605650 27833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.67.keytab
--principal=kudu/127.8.168.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:34211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:38.606441 27833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:38.607223 27833 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:38.614959 27838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:38.615144 27841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:38.615551 27839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:38.616102 27833 server_base.cc:1034] running on GCE node
I20240501 20:01:38.616425 27833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:38.616933 27833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:38.618086 27833 hybrid_clock.cc:648] HybridClock initialized: now 1714593698618075 us; error 38 us; skew 500 ppm
May 01 20:01:38 dist-test-slave-tzct krb5kdc[27458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593698, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:38.620648 27833 init.cc:377] Logged in from keytab as kudu/127.8.168.67@KRBTEST.COM (short username kudu)
I20240501 20:01:38.621502 27833 webserver.cc:460] Webserver started at http://127.8.168.67:43359/ using document root <none> and password file <none>
I20240501 20:01:38.621970 27833 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:38.622018 27833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:38.622152 27833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:38.623785 27833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "e64ccea3a359494ca485598ecccc2f32"
format_stamp: "Formatted at 2024-05-01 20:01:38 on dist-test-slave-tzct"
server_key: "e5fa0d6640f8500ba403b5a024c14cdf"
server_key_iv: "72f4e4492bd4f407e46cedcb25993455"
server_key_version: "encryptionkey@0"
I20240501 20:01:38.624204 27833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "e64ccea3a359494ca485598ecccc2f32"
format_stamp: "Formatted at 2024-05-01 20:01:38 on dist-test-slave-tzct"
server_key: "e5fa0d6640f8500ba403b5a024c14cdf"
server_key_iv: "72f4e4492bd4f407e46cedcb25993455"
server_key_version: "encryptionkey@0"
I20240501 20:01:38.627406 27833 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 20:01:38.629549 27848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:38.630578 27833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:38.630674 27833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "e64ccea3a359494ca485598ecccc2f32"
format_stamp: "Formatted at 2024-05-01 20:01:38 on dist-test-slave-tzct"
server_key: "e5fa0d6640f8500ba403b5a024c14cdf"
server_key_iv: "72f4e4492bd4f407e46cedcb25993455"
server_key_version: "encryptionkey@0"
I20240501 20:01:38.630751 27833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:38.649432 27833 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:38.652313 27833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:38.652455 27833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:38.652999 27833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:38.653749 27833 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:38.653806 27833 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:38.653846 27833 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:38.653867 27833 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:38.661787 27833 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:37211
I20240501 20:01:38.661863 27961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:37211 every 8 connection(s)
I20240501 20:01:38.662669 27833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 20:01:38.666322 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 27833
I20240501 20:01:38.666443 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 20:01:38.666688 8865 external_mini_cluster.cc:1428] Setting key cfd0274c6ad27a218e299f8a0eeb66f5
May 01 20:01:38 dist-test-slave-tzct krb5kdc[27458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593698, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.67@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:01:38.674311 27542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:38.664247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:43145 (local address 127.8.168.126:34211)
0501 20:01:38.664421 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:38.664425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:38.665043 (+ 618us) server_negotiation.cc:407] Connection header received
0501 20:01:38.665775 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:38.665778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:38.665825 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:38.665899 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:38.667983 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.668496 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:38.669206 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.669381 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:38.671320 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:38.671334 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:38.671336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:38.671360 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:38.672615 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:38.673078 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:38.673081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:38.673082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:38.673125 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:38.673367 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:38.673369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:38.673370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:38.673509 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:38.673590 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:38.674048 (+ 458us) server_negotiation.cc:299] Negotiation successful
0501 20:01:38.674162 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20240501 20:01:38.674860 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:38.664510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:34211 (local address 127.8.168.67:43145)
0501 20:01:38.664918 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:38.664953 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:38.665605 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:38.666036 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:38.666043 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:38.666536 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:38.667784 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:38.667849 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.668618 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:38.668621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:38.669095 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:38.669105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.669301 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:38.669858 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:01:38.669881 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:38.671181 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:38.672745 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:38.672750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:38.672760 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:38.672988 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:38.673200 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:01:38.673203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:01:38.673205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:01:38.673278 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:01:38.673604 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:38.673610 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:38.673814 (+ 204us) client_negotiation.cc:769] Sending connection context
0501 20:01:38.673980 (+ 166us) client_negotiation.cc:240] Negotiation successful
0501 20:01:38.674224 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":97,"threads_started":1}
I20240501 20:01:38.675745 27962 heartbeater.cc:347] Connected to a master server at 127.8.168.126:34211
I20240501 20:01:38.675944 27962 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:38.676396 27962 heartbeater.cc:511] Master 127.8.168.126:34211 requested a full tablet report, sending...
I20240501 20:01:38.677294 27503 ts_manager.cc:194] Registered new tserver with Master: e64ccea3a359494ca485598ecccc2f32 (127.8.168.67:37211)
I20240501 20:01:38.677816 27503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.168.67@KRBTEST.COM'} at 127.8.168.67:43145
I20240501 20:01:38.680351 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 20:01:38.687875 27542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:38.682197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33878 (local address 127.8.168.126:34211)
0501 20:01:38.682322 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:38.682325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:38.682451 (+ 126us) server_negotiation.cc:407] Connection header received
0501 20:01:38.682558 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:38.682560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:38.682596 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:38.682655 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:38.683418 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.683866 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:38.684455 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.684565 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:38.685518 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:38.685530 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:38.685532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:38.685552 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:38.686776 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:38.687089 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:38.687091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:38.687092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:38.687125 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:38.687262 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:38.687264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:38.687265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:38.687413 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:38.687511 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:38.687649 (+ 138us) server_negotiation.cc:299] Negotiation successful
0501 20:01:38.687730 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20240501 20:01:38.690140 27503 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:01:38.692411 27503 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:01:38.703619 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:38.699565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:40681 (local address 127.0.0.1:37006)
0501 20:01:38.699857 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:38.699923 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:38.700043 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:38.701169 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:38.701175 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:38.701194 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:38.701448 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:38.701456 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.702462 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:38.702465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:38.703163 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:38.703172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.703269 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:38.703329 (+ 60us) client_negotiation.cc:769] Sending connection context
0501 20:01:38.703413 (+ 84us) client_negotiation.cc:240] Negotiation successful
0501 20:01:38.703495 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":68,"threads_started":1}
I20240501 20:01:38.704033 27974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:38.699653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:37211 (local address 127.0.0.1:52362)
0501 20:01:38.700302 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:38.700315 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:38.700403 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:38.701352 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:38.701355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:38.701371 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:38.701573 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:38.701580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.702875 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:38.702879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:38.703678 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:38.703688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.703795 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:38.703811 (+ 16us) client_negotiation.cc:769] Sending connection context
0501 20:01:38.703869 (+ 58us) client_negotiation.cc:240] Negotiation successful
0501 20:01:38.703922 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":86,"threads_started":1}
I20240501 20:01:38.704082 27975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:38.699907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:39369 (local address 127.0.0.1:35462)
0501 20:01:38.700375 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:38.700399 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:38.700473 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:38.701496 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:38.701498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:38.701515 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:38.701745 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:38.701752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.703055 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:38.703059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:38.703810 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:38.703820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.703899 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:38.703915 (+ 16us) client_negotiation.cc:769] Sending connection context
0501 20:01:38.703943 (+ 28us) client_negotiation.cc:240] Negotiation successful
0501 20:01:38.703984 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":95,"threads_started":1}
I20240501 20:01:38.704867 27976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:38.700160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.8.168.66:40681)
0501 20:01:38.700758 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:38.700765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:38.700788 (+ 23us) server_negotiation.cc:407] Connection header received
0501 20:01:38.700877 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:38.700881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:38.701018 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:38.701152 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:38.701577 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.702358 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:38.703917 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.704450 (+ 533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:38.704562 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:38.704629 (+ 67us) server_negotiation.cc:299] Negotiation successful
0501 20:01:38.704705 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":88,"threads_started":1}
I20240501 20:01:38.705082 27978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:38.700581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.8.168.65:39369)
0501 20:01:38.700901 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:38.700907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:38.700929 (+ 22us) server_negotiation.cc:407] Connection header received
0501 20:01:38.701011 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:38.701016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:38.701178 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:38.701361 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:38.701863 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.702933 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:38.704356 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.704714 (+ 358us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:38.704827 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:38.704881 (+ 54us) server_negotiation.cc:299] Negotiation successful
0501 20:01:38.704949 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":142,"threads_started":1}
I20240501 20:01:38.705158 27977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:38.700455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52362 (local address 127.8.168.67:37211)
0501 20:01:38.700963 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:38.700968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:38.700986 (+ 18us) server_negotiation.cc:407] Connection header received
0501 20:01:38.701052 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:38.701056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:38.701177 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:38.701361 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:38.701677 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.702765 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:38.704280 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:38.704780 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:38.704889 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:38.704954 (+ 65us) server_negotiation.cc:299] Negotiation successful
0501 20:01:38.705034 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":118,"threads_started":1}
I20240501 20:01:38.707194 27896 tablet_service.cc:1467] Processing CreateTablet for tablet 6efee26cf9654057b192283362527b3b (DEFAULT_TABLE table=test-table [id=2d5cef453b064f6ca58de34d92dd62ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:38.707208 27628 tablet_service.cc:1467] Processing CreateTablet for tablet 6efee26cf9654057b192283362527b3b (DEFAULT_TABLE table=test-table [id=2d5cef453b064f6ca58de34d92dd62ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:38.707207 27762 tablet_service.cc:1467] Processing CreateTablet for tablet 6efee26cf9654057b192283362527b3b (DEFAULT_TABLE table=test-table [id=2d5cef453b064f6ca58de34d92dd62ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:38.707989 27896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6efee26cf9654057b192283362527b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:38.708005 27762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6efee26cf9654057b192283362527b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:38.708005 27628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6efee26cf9654057b192283362527b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:38.713132 27979 tablet_bootstrap.cc:492] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32: Bootstrap starting.
I20240501 20:01:38.714417 27980 tablet_bootstrap.cc:492] T 6efee26cf9654057b192283362527b3b P fdd3e9d4f4174f05b85cfb1ac8f0135b: Bootstrap starting.
I20240501 20:01:38.714766 27981 tablet_bootstrap.cc:492] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c: Bootstrap starting.
I20240501 20:01:38.715173 27979 tablet_bootstrap.cc:654] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:38.715822 27979 log.cc:824] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:38.716351 27981 tablet_bootstrap.cc:654] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:38.717034 27981 log.cc:824] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:38.717418 27979 tablet_bootstrap.cc:492] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32: No bootstrap required, opened a new log
I20240501 20:01:38.717598 27979 ts_tablet_manager.cc:1367] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:01:38.718564 27980 tablet_bootstrap.cc:654] T 6efee26cf9654057b192283362527b3b P fdd3e9d4f4174f05b85cfb1ac8f0135b: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:38.718730 27981 tablet_bootstrap.cc:492] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c: No bootstrap required, opened a new log
I20240501 20:01:38.718894 27981 ts_tablet_manager.cc:1367] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:01:38.719326 27980 log.cc:824] T 6efee26cf9654057b192283362527b3b P fdd3e9d4f4174f05b85cfb1ac8f0135b: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:38.720448 27979 raft_consensus.cc:348] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10d9e66d056437db3a86bcc8b98436c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 40681 } } peers { permanent_uuid: "e64ccea3a359494ca485598ecccc2f32" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37211 } } peers { permanent_uuid: "fdd3e9d4f4174f05b85cfb1ac8f0135b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39369 } }
I20240501 20:01:38.720762 27979 raft_consensus.cc:374] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:38.720867 27979 raft_consensus.cc:729] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64ccea3a359494ca485598ecccc2f32, State: Initialized, Role: FOLLOWER
I20240501 20:01:38.721266 27979 consensus_queue.cc:260] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10d9e66d056437db3a86bcc8b98436c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 40681 } } peers { permanent_uuid: "e64ccea3a359494ca485598ecccc2f32" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37211 } } peers { permanent_uuid: "fdd3e9d4f4174f05b85cfb1ac8f0135b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39369 } }
I20240501 20:01:38.721460 27980 tablet_bootstrap.cc:492] T 6efee26cf9654057b192283362527b3b P fdd3e9d4f4174f05b85cfb1ac8f0135b: No bootstrap required, opened a new log
I20240501 20:01:38.721624 27980 ts_tablet_manager.cc:1367] T 6efee26cf9654057b192283362527b3b P fdd3e9d4f4174f05b85cfb1ac8f0135b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:01:38.721975 27979 ts_tablet_manager.cc:1398] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 20:01:38.722165 27981 raft_consensus.cc:348] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10d9e66d056437db3a86bcc8b98436c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 40681 } } peers { permanent_uuid: "e64ccea3a359494ca485598ecccc2f32" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37211 } } peers { permanent_uuid: "fdd3e9d4f4174f05b85cfb1ac8f0135b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39369 } }
I20240501 20:01:38.722309 27962 heartbeater.cc:503] Master 127.8.168.126:34211 was elected leader, sending a full tablet report...
I20240501 20:01:38.722427 27981 raft_consensus.cc:374] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:38.722493 27981 raft_consensus.cc:729] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10d9e66d056437db3a86bcc8b98436c, State: Initialized, Role: FOLLOWER
I20240501 20:01:38.722949 27981 consensus_queue.cc:260] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10d9e66d056437db3a86bcc8b98436c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 40681 } } peers { permanent_uuid: "e64ccea3a359494ca485598ecccc2f32" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37211 } } peers { permanent_uuid: "fdd3e9d4f4174f05b85cfb1ac8f0135b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39369 } }
I20240501 20:01:38.723557 27828 heartbeater.cc:503] Master 127.8.168.126:34211 was elected leader, sending a full tablet report...
I20240501 20:01:38.723732 27981 ts_tablet_manager.cc:1398] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 20:01:38.725071 27980 raft_consensus.cc:348] T 6efee26cf9654057b192283362527b3b P fdd3e9d4f4174f05b85cfb1ac8f0135b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10d9e66d056437db3a86bcc8b98436c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 40681 } } peers { permanent_uuid: "e64ccea3a359494ca485598ecccc2f32" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37211 } } peers { permanent_uuid: "fdd3e9d4f4174f05b85cfb1ac8f0135b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39369 } }
I20240501 20:01:38.725239 27980 raft_consensus.cc:374] T 6efee26cf9654057b192283362527b3b P fdd3e9d4f4174f05b85cfb1ac8f0135b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:38.725314 27980 raft_consensus.cc:729] T 6efee26cf9654057b192283362527b3b P fdd3e9d4f4174f05b85cfb1ac8f0135b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd3e9d4f4174f05b85cfb1ac8f0135b, State: Initialized, Role: FOLLOWER
I20240501 20:01:38.725714 27980 consensus_queue.cc:260] T 6efee26cf9654057b192283362527b3b P fdd3e9d4f4174f05b85cfb1ac8f0135b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10d9e66d056437db3a86bcc8b98436c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 40681 } } peers { permanent_uuid: "e64ccea3a359494ca485598ecccc2f32" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37211 } } peers { permanent_uuid: "fdd3e9d4f4174f05b85cfb1ac8f0135b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39369 } }
I20240501 20:01:38.726366 27694 heartbeater.cc:503] Master 127.8.168.126:34211 was elected leader, sending a full tablet report...
I20240501 20:01:38.726440 27980 ts_tablet_manager.cc:1398] T 6efee26cf9654057b192283362527b3b P fdd3e9d4f4174f05b85cfb1ac8f0135b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20240501 20:01:38.756522 27695 tablet.cc:2369] T 6efee26cf9654057b192283362527b3b P fdd3e9d4f4174f05b85cfb1ac8f0135b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:01:38.913493 27963 tablet.cc:2369] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:01:38.962194 27829 tablet.cc:2369] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:01:39.045132 27985 raft_consensus.cc:482] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:39.045325 27985 raft_consensus.cc:504] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10d9e66d056437db3a86bcc8b98436c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 40681 } } peers { permanent_uuid: "e64ccea3a359494ca485598ecccc2f32" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37211 } } peers { permanent_uuid: "fdd3e9d4f4174f05b85cfb1ac8f0135b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39369 } }
I20240501 20:01:39.046468 27985 leader_election.cc:288] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f10d9e66d056437db3a86bcc8b98436c (127.8.168.66:40681), fdd3e9d4f4174f05b85cfb1ac8f0135b (127.8.168.65:39369)
I20240501 20:01:39.049201 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.046716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:40681 (local address 127.8.168.67:41705)
0501 20:01:39.046917 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:39.046932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:39.047031 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:39.047271 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:39.047273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:39.047290 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:39.047492 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:39.047497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.048215 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.048218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:39.048821 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:39.048830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.048937 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.048960 (+ 23us) client_negotiation.cc:769] Sending connection context
0501 20:01:39.049013 (+ 53us) client_negotiation.cc:240] Negotiation successful
0501 20:01:39.049071 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20240501 20:01:39.049675 27976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.046809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:41705 (local address 127.8.168.66:40681)
0501 20:01:39.046937 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:39.046941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:39.046979 (+ 38us) server_negotiation.cc:407] Connection header received
0501 20:01:39.047136 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:39.047139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:39.047181 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:39.047243 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:39.047604 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.048106 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.048946 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.049408 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.049449 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:39.049501 (+ 52us) server_negotiation.cc:299] Negotiation successful
0501 20:01:39.049560 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20240501 20:01:39.049731 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.046848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:39369 (local address 127.8.168.67:45621)
0501 20:01:39.047213 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:39.047225 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:39.047379 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:39.047658 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:39.047661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:39.047679 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:39.047923 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:39.047930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.048664 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.048667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:39.049331 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:39.049344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.049506 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.049525 (+ 19us) client_negotiation.cc:769] Sending connection context
0501 20:01:39.049568 (+ 43us) client_negotiation.cc:240] Negotiation successful
0501 20:01:39.049613 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":124,"threads_started":1}
I20240501 20:01:39.050182 27978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.046887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:45621 (local address 127.8.168.65:39369)
0501 20:01:39.047029 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:39.047033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:39.047320 (+ 287us) server_negotiation.cc:407] Connection header received
0501 20:01:39.047502 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:39.047504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:39.047563 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:39.047633 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:39.048050 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.048530 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.049512 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.049962 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.050000 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:39.050049 (+ 49us) server_negotiation.cc:299] Negotiation successful
0501 20:01:39.050094 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20240501 20:01:39.050297 27782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6efee26cf9654057b192283362527b3b" candidate_uuid: "e64ccea3a359494ca485598ecccc2f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10d9e66d056437db3a86bcc8b98436c" is_pre_election: true
I20240501 20:01:39.050591 27782 raft_consensus.cc:2454] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e64ccea3a359494ca485598ecccc2f32 in term 0.
I20240501 20:01:39.050653 27648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6efee26cf9654057b192283362527b3b" candidate_uuid: "e64ccea3a359494ca485598ecccc2f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd3e9d4f4174f05b85cfb1ac8f0135b" is_pre_election: true
I20240501 20:01:39.050905 27648 raft_consensus.cc:2454] T 6efee26cf9654057b192283362527b3b P fdd3e9d4f4174f05b85cfb1ac8f0135b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e64ccea3a359494ca485598ecccc2f32 in term 0.
I20240501 20:01:39.051116 27851 leader_election.cc:302] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e64ccea3a359494ca485598ecccc2f32, f10d9e66d056437db3a86bcc8b98436c; no voters:
I20240501 20:01:39.051393 27985 raft_consensus.cc:2781] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:39.051457 27985 raft_consensus.cc:482] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:39.051486 27985 raft_consensus.cc:3037] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:39.052346 27985 raft_consensus.cc:504] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10d9e66d056437db3a86bcc8b98436c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 40681 } } peers { permanent_uuid: "e64ccea3a359494ca485598ecccc2f32" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37211 } } peers { permanent_uuid: "fdd3e9d4f4174f05b85cfb1ac8f0135b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39369 } }
I20240501 20:01:39.052666 27985 leader_election.cc:288] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [CANDIDATE]: Term 1 election: Requested vote from peers f10d9e66d056437db3a86bcc8b98436c (127.8.168.66:40681), fdd3e9d4f4174f05b85cfb1ac8f0135b (127.8.168.65:39369)
I20240501 20:01:39.053103 27648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6efee26cf9654057b192283362527b3b" candidate_uuid: "e64ccea3a359494ca485598ecccc2f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd3e9d4f4174f05b85cfb1ac8f0135b"
I20240501 20:01:39.053161 27782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6efee26cf9654057b192283362527b3b" candidate_uuid: "e64ccea3a359494ca485598ecccc2f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10d9e66d056437db3a86bcc8b98436c"
I20240501 20:01:39.053226 27648 raft_consensus.cc:3037] T 6efee26cf9654057b192283362527b3b P fdd3e9d4f4174f05b85cfb1ac8f0135b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:39.053272 27782 raft_consensus.cc:3037] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:39.054436 27782 raft_consensus.cc:2454] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e64ccea3a359494ca485598ecccc2f32 in term 1.
I20240501 20:01:39.054436 27648 raft_consensus.cc:2454] T 6efee26cf9654057b192283362527b3b P fdd3e9d4f4174f05b85cfb1ac8f0135b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e64ccea3a359494ca485598ecccc2f32 in term 1.
I20240501 20:01:39.054759 27852 leader_election.cc:302] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e64ccea3a359494ca485598ecccc2f32, fdd3e9d4f4174f05b85cfb1ac8f0135b; no voters:
I20240501 20:01:39.054934 27985 raft_consensus.cc:2781] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:39.055122 27985 raft_consensus.cc:686] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [term 1 LEADER]: Becoming Leader. State: Replica: e64ccea3a359494ca485598ecccc2f32, State: Running, Role: LEADER
I20240501 20:01:39.055402 27985 consensus_queue.cc:237] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10d9e66d056437db3a86bcc8b98436c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 40681 } } peers { permanent_uuid: "e64ccea3a359494ca485598ecccc2f32" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37211 } } peers { permanent_uuid: "fdd3e9d4f4174f05b85cfb1ac8f0135b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39369 } }
I20240501 20:01:39.058283 27503 catalog_manager.cc:5484] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 reported cstate change: term changed from 0 to 1, leader changed from <none> to e64ccea3a359494ca485598ecccc2f32 (127.8.168.67). New cstate: current_term: 1 leader_uuid: "e64ccea3a359494ca485598ecccc2f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f10d9e66d056437db3a86bcc8b98436c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 40681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e64ccea3a359494ca485598ecccc2f32" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 37211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdd3e9d4f4174f05b85cfb1ac8f0135b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:39.151487 27977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.148700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52376 (local address 127.8.168.67:37211)
0501 20:01:39.148888 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:39.148891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:39.148905 (+ 14us) server_negotiation.cc:407] Connection header received
0501 20:01:39.148982 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:39.148984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:39.149036 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:39.149137 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:01:39.149414 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.149939 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.150493 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.150628 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.150712 (+ 84us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:01:39.151150 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:01:39.151238 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:39.151302 (+ 64us) server_negotiation.cc:299] Negotiation successful
0501 20:01:39.151353 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20240501 20:01:39.156939 27648 raft_consensus.cc:1261] T 6efee26cf9654057b192283362527b3b P fdd3e9d4f4174f05b85cfb1ac8f0135b [term 1 FOLLOWER]: Refusing update from remote peer e64ccea3a359494ca485598ecccc2f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:39.156939 27782 raft_consensus.cc:1261] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c [term 1 FOLLOWER]: Refusing update from remote peer e64ccea3a359494ca485598ecccc2f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:39.157670 27989 consensus_queue.cc:1035] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdd3e9d4f4174f05b85cfb1ac8f0135b" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:39.157845 27985 consensus_queue.cc:1035] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f10d9e66d056437db3a86bcc8b98436c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 40681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:39.165270 27994 mvcc.cc:205] Tried to move back new op lower bound from 7022975791739531264 to 7022975791333285888. Current Snapshot: MvccSnapshot[applied={T|T < 7022975791739531264}]
I20240501 20:01:39.166086 27993 mvcc.cc:205] Tried to move back new op lower bound from 7022975791739531264 to 7022975791333285888. Current Snapshot: MvccSnapshot[applied={T|T < 7022975791739531264}]
I20240501 20:01:39.172144 27501 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33878:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:01:39.172284 27501 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33878:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:01:39.174386 27501 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 280d641558804180805d8b524f1a8746: Sending DeleteTablet for 3 replicas of tablet 6efee26cf9654057b192283362527b3b
I20240501 20:01:39.175063 27762 tablet_service.cc:1514] Processing DeleteTablet for tablet 6efee26cf9654057b192283362527b3b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:39 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:37006
I20240501 20:01:39.175065 27896 tablet_service.cc:1514] Processing DeleteTablet for tablet 6efee26cf9654057b192283362527b3b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:39 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:52362
I20240501 20:01:39.175525 28003 tablet_replica.cc:330] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c: stopping tablet replica
I20240501 20:01:39.175729 27628 tablet_service.cc:1514] Processing DeleteTablet for tablet 6efee26cf9654057b192283362527b3b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:39 UTC) from {username='kudu', principal='kudu/127.8.168.126@KRBTEST.COM'} at 127.0.0.1:35462
I20240501 20:01:39.175823 28003 raft_consensus.cc:2229] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:39.176123 28003 raft_consensus.cc:2258] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:39.176141 28004 tablet_replica.cc:330] T 6efee26cf9654057b192283362527b3b P fdd3e9d4f4174f05b85cfb1ac8f0135b: stopping tablet replica
I20240501 20:01:39.176458 28002 tablet_replica.cc:330] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32: stopping tablet replica
I20240501 20:01:39.176539 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 27565
I20240501 20:01:39.177031 28002 raft_consensus.cc:2229] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:39.177335 28002 raft_consensus.cc:2258] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:39.177484 28003 ts_tablet_manager.cc:1875] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:01:39.178486 28002 ts_tablet_manager.cc:1875] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:01:39.179844 28003 ts_tablet_manager.cc:1888] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:01:39.179932 28003 log.cc:1196] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/6efee26cf9654057b192283362527b3b
I20240501 20:01:39.180233 28003 ts_tablet_manager.cc:1909] T 6efee26cf9654057b192283362527b3b P f10d9e66d056437db3a86bcc8b98436c: Deleting consensus metadata
I20240501 20:01:39.180840 28002 ts_tablet_manager.cc:1888] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:01:39.180953 28002 log.cc:1196] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/6efee26cf9654057b192283362527b3b
I20240501 20:01:39.181270 28002 ts_tablet_manager.cc:1909] T 6efee26cf9654057b192283362527b3b P e64ccea3a359494ca485598ecccc2f32: Deleting consensus metadata
I20240501 20:01:39.181351 27489 catalog_manager.cc:4830] TS f10d9e66d056437db3a86bcc8b98436c (127.8.168.66:40681): tablet 6efee26cf9654057b192283362527b3b (table test-table [id=2d5cef453b064f6ca58de34d92dd62ca]) successfully deleted
I20240501 20:01:39.182328 27487 catalog_manager.cc:4830] TS e64ccea3a359494ca485598ecccc2f32 (127.8.168.67:37211): tablet 6efee26cf9654057b192283362527b3b (table test-table [id=2d5cef453b064f6ca58de34d92dd62ca]) successfully deleted
W20240501 20:01:39.184597 27490 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.168.65:39369 (error 108)
I20240501 20:01:39.185274 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 27699
I20240501 20:01:39.185544 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.185286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:39369 (local address 127.0.0.1:35464)
0501 20:01:39.185410 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:39.185464 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:39369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20240501 20:01:39.186895 27490 catalog_manager.cc:4573] TS fdd3e9d4f4174f05b85cfb1ac8f0135b (127.8.168.65:39369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6efee26cf9654057b192283362527b3b: Network error: Client connection negotiation failed: client connection to 127.8.168.65:39369: connect: Connection refused (error 111)
I20240501 20:01:39.190310 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 27833
I20240501 20:01:39.195242 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 27471
2024-05-01T20:01:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2384 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2024-05-01T20:01:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:39Z Disabled control of system clock
I20240501 20:01:39.221418 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33789
--webserver_interface=127.8.168.126
--webserver_port=0
--builtin_ntp_servers=127.8.168.84:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:33789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240501 20:01:39.324935 28009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:39.325171 28009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:39.325218 28009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:39.328212 28009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:39.328274 28009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:39.328287 28009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:39.328305 28009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:39.328320 28009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:39.331918 28009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:33789
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:39.332784 28009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:39.333884 28009 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:39.340777 28017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:39.341130 28014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:39.341615 28015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:39.342262 28009 server_base.cc:1034] running on GCE node
I20240501 20:01:39.342828 28009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:39.343794 28009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:39.344957 28009 hybrid_clock.cc:648] HybridClock initialized: now 1714593699344947 us; error 27 us; skew 500 ppm
I20240501 20:01:39.346889 28009 webserver.cc:460] Webserver started at http://127.8.168.126:38399/ using document root <none> and password file <none>
I20240501 20:01:39.347445 28009 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:39.347503 28009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:39.347702 28009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:39.349606 28009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "d537bc35eac543d4afb06ba874bf5361"
format_stamp: "Formatted at 2024-05-01 20:01:39 on dist-test-slave-tzct"
server_key: "27b7f176ab72b1adc1a0d06b686ab340"
server_key_iv: "48d57452eeb5a435b48e11b466d712dd"
server_key_version: "encryptionkey@0"
I20240501 20:01:39.350103 28009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "d537bc35eac543d4afb06ba874bf5361"
format_stamp: "Formatted at 2024-05-01 20:01:39 on dist-test-slave-tzct"
server_key: "27b7f176ab72b1adc1a0d06b686ab340"
server_key_iv: "48d57452eeb5a435b48e11b466d712dd"
server_key_version: "encryptionkey@0"
I20240501 20:01:39.353539 28009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:01:39.355681 28023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:39.356616 28009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:39.356755 28009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "d537bc35eac543d4afb06ba874bf5361"
format_stamp: "Formatted at 2024-05-01 20:01:39 on dist-test-slave-tzct"
server_key: "27b7f176ab72b1adc1a0d06b686ab340"
server_key_iv: "48d57452eeb5a435b48e11b466d712dd"
server_key_version: "encryptionkey@0"
I20240501 20:01:39.356848 28009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:39.370608 28009 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:39.371238 28009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:39.371405 28009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:39.377952 28009 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:33789
I20240501 20:01:39.377966 28075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:33789 every 8 connection(s)
I20240501 20:01:39.378885 28009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:39.381413 28076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:39.386039 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 28009
I20240501 20:01:39.386159 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:39.386363 8865 external_mini_cluster.cc:1428] Setting key 0d9ddb5c81589b87eb8afa414240996a
I20240501 20:01:39.386626 28076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361: Bootstrap starting.
I20240501 20:01:39.388715 28076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:39.389432 28076 log.cc:824] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:39.391178 28076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361: No bootstrap required, opened a new log
I20240501 20:01:39.393934 28076 raft_consensus.cc:348] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d537bc35eac543d4afb06ba874bf5361" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33789 } }
I20240501 20:01:39.394104 28076 raft_consensus.cc:374] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:39.394147 28076 raft_consensus.cc:729] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d537bc35eac543d4afb06ba874bf5361, State: Initialized, Role: FOLLOWER
I20240501 20:01:39.394265 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.387561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.8.168.126:33789)
0501 20:01:39.388138 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:39.388145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:39.388178 (+ 33us) server_negotiation.cc:407] Connection header received
0501 20:01:39.388771 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:39.388789 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:39.389136 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:39.389444 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:39.389963 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.390772 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.391647 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.391853 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.392402 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:39.392419 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:39.392429 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:39.392811 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:39.392862 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:39.392879 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:39.392960 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:39.393099 (+ 139us) server_negotiation.cc:299] Negotiation successful
0501 20:01:39.393351 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":110,"threads_started":1}
I20240501 20:01:39.394610 28076 consensus_queue.cc:260] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d537bc35eac543d4afb06ba874bf5361" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33789 } }
I20240501 20:01:39.394760 28076 raft_consensus.cc:388] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:39.394821 28076 raft_consensus.cc:482] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:39.394918 28076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:39.395689 28076 raft_consensus.cc:504] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d537bc35eac543d4afb06ba874bf5361" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33789 } }
I20240501 20:01:39.396003 28076 leader_election.cc:302] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d537bc35eac543d4afb06ba874bf5361; no voters:
I20240501 20:01:39.396303 28076 leader_election.cc:288] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:39.396411 28081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:39.396600 28081 raft_consensus.cc:686] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [term 1 LEADER]: Becoming Leader. State: Replica: d537bc35eac543d4afb06ba874bf5361, State: Running, Role: LEADER
I20240501 20:01:39.397043 28081 consensus_queue.cc:237] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d537bc35eac543d4afb06ba874bf5361" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33789 } }
I20240501 20:01:39.397281 28076 sys_catalog.cc:564] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:39.398696 28082 sys_catalog.cc:455] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d537bc35eac543d4afb06ba874bf5361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d537bc35eac543d4afb06ba874bf5361" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33789 } } }
I20240501 20:01:39.398694 28083 sys_catalog.cc:455] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d537bc35eac543d4afb06ba874bf5361. Latest consensus state: current_term: 1 leader_uuid: "d537bc35eac543d4afb06ba874bf5361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d537bc35eac543d4afb06ba874bf5361" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33789 } } }
I20240501 20:01:39.398844 28083 sys_catalog.cc:458] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:39.398844 28082 sys_catalog.cc:458] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361 [sys.catalog]: This master's current role is: LEADER
W20240501 20:01:39.401562 28096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:01:39.401664 28096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:01:39.401759 28095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:39.402771 28095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:39.407670 28095 catalog_manager.cc:1347] Generated new cluster ID: c2262baf02a145ad83b42715bc2e158a
I20240501 20:01:39.407734 28095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:39.429996 28095 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:39.430986 28095 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:39.436239 28095 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361: Generated new TSK 0
I20240501 20:01:39.436973 28095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:39.443671 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.65:0
--local_ip_for_outbound_sockets=127.8.168.65
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33789
--builtin_ntp_servers=127.8.168.84:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240501 20:01:39.549402 28100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:39.549629 28100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:39.549679 28100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:39.552807 28100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:39.552862 28100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:39.552943 28100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.65
I20240501 20:01:39.556548 28100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.168.65
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.65
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:39.557484 28100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:39.558648 28100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:39.566937 28108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:39.567032 28105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:39.567405 28106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:39.567940 28100 server_base.cc:1034] running on GCE node
I20240501 20:01:39.568310 28100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:39.568912 28100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:39.570062 28100 hybrid_clock.cc:648] HybridClock initialized: now 1714593699570047 us; error 30 us; skew 500 ppm
I20240501 20:01:39.571758 28100 webserver.cc:460] Webserver started at http://127.8.168.65:33521/ using document root <none> and password file <none>
I20240501 20:01:39.572284 28100 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:39.572348 28100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:39.572556 28100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:39.574898 28100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-0/data/instance:
uuid: "e090c185833a4b938dbda0a99700cc20"
format_stamp: "Formatted at 2024-05-01 20:01:39 on dist-test-slave-tzct"
server_key: "085d4aa498523fdbd665dad86d395e96"
server_key_iv: "b28270d0de6055d7cd60142f70f0952a"
server_key_version: "encryptionkey@0"
I20240501 20:01:39.575486 28100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance:
uuid: "e090c185833a4b938dbda0a99700cc20"
format_stamp: "Formatted at 2024-05-01 20:01:39 on dist-test-slave-tzct"
server_key: "085d4aa498523fdbd665dad86d395e96"
server_key_iv: "b28270d0de6055d7cd60142f70f0952a"
server_key_version: "encryptionkey@0"
I20240501 20:01:39.578894 28100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 20:01:39.581108 28114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:39.582157 28100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:39.582262 28100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-0/wal
uuid: "e090c185833a4b938dbda0a99700cc20"
format_stamp: "Formatted at 2024-05-01 20:01:39 on dist-test-slave-tzct"
server_key: "085d4aa498523fdbd665dad86d395e96"
server_key_iv: "b28270d0de6055d7cd60142f70f0952a"
server_key_version: "encryptionkey@0"
I20240501 20:01:39.582348 28100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:39.594961 28100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:39.595597 28100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:39.595748 28100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:39.596262 28100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:39.597185 28100 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:39.597232 28100 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:39.597282 28100 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:39.597308 28100 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:39.605561 28100 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.65:39483
I20240501 20:01:39.605618 28227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.65:39483 every 8 connection(s)
I20240501 20:01:39.606446 28100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-0/data/info.pb
I20240501 20:01:39.608986 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 28100
I20240501 20:01:39.609103 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-0/wal/instance
I20240501 20:01:39.609320 8865 external_mini_cluster.cc:1428] Setting key 2277608eb27815f1fc4ff0f2471374bc
I20240501 20:01:39.611163 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.66:0
--local_ip_for_outbound_sockets=127.8.168.66
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33789
--builtin_ntp_servers=127.8.168.84:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240501 20:01:39.613960 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.608017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.65:48843 (local address 127.8.168.126:33789)
0501 20:01:39.608135 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:39.608139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:39.609046 (+ 907us) server_negotiation.cc:407] Connection header received
0501 20:01:39.609842 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:39.609846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:39.609913 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:39.609998 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:39.611404 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.611901 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.612505 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.612684 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.613428 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:39.613454 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:39.613457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:39.613469 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:39.613489 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:39.613495 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:39.613554 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:39.613727 (+ 173us) server_negotiation.cc:299] Negotiation successful
0501 20:01:39.613800 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240501 20:01:39.614507 28230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.608252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33789 (local address 127.8.168.65:48843)
0501 20:01:39.608913 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:39.608946 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:39.609642 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:39.610146 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:39.610154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:39.610225 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:39.611222 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:39.611242 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.612017 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.612020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:39.612396 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:39.612403 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.612602 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.613209 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 20:01:39.613211 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:39.613245 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 20:01:39.613300 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:39.613565 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:39.613572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:39.613578 (+ 6us) client_negotiation.cc:769] Sending connection context
0501 20:01:39.613692 (+ 114us) client_negotiation.cc:240] Negotiation successful
0501 20:01:39.613895 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":97,"threads_started":1}
I20240501 20:01:39.615743 28228 heartbeater.cc:347] Connected to a master server at 127.8.168.126:33789
I20240501 20:01:39.615994 28228 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:39.616672 28228 heartbeater.cc:511] Master 127.8.168.126:33789 requested a full tablet report, sending...
I20240501 20:01:39.618271 28040 ts_manager.cc:194] Registered new tserver with Master: e090c185833a4b938dbda0a99700cc20 (127.8.168.65:39483)
I20240501 20:01:39.619450 28040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.168.65:48843
W20240501 20:01:39.715566 28231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:39.715837 28231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:39.715911 28231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:39.719318 28231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:39.719374 28231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:39.719458 28231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.66
I20240501 20:01:39.723062 28231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.168.66
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.66
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:39.723906 28231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:39.725030 28231 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:39.732698 28236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:39.733109 28239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:39.733529 28237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:39.734014 28231 server_base.cc:1034] running on GCE node
I20240501 20:01:39.734356 28231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:39.734895 28231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:39.736037 28231 hybrid_clock.cc:648] HybridClock initialized: now 1714593699736022 us; error 28 us; skew 500 ppm
I20240501 20:01:39.737747 28231 webserver.cc:460] Webserver started at http://127.8.168.66:40165/ using document root <none> and password file <none>
I20240501 20:01:39.738276 28231 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:39.738329 28231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:39.738499 28231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:39.740192 28231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/data/instance:
uuid: "3e7a7fcfab5e482588f4aff3e485175c"
format_stamp: "Formatted at 2024-05-01 20:01:39 on dist-test-slave-tzct"
server_key: "a1dc32e0d413270770d775685087b4ac"
server_key_iv: "3ddb0fbff15270d8b8641f5f585217fa"
server_key_version: "encryptionkey@0"
I20240501 20:01:39.740634 28231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance:
uuid: "3e7a7fcfab5e482588f4aff3e485175c"
format_stamp: "Formatted at 2024-05-01 20:01:39 on dist-test-slave-tzct"
server_key: "a1dc32e0d413270770d775685087b4ac"
server_key_iv: "3ddb0fbff15270d8b8641f5f585217fa"
server_key_version: "encryptionkey@0"
I20240501 20:01:39.743692 28231 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240501 20:01:39.745815 28245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:39.746754 28231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:01:39.746846 28231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/wal
uuid: "3e7a7fcfab5e482588f4aff3e485175c"
format_stamp: "Formatted at 2024-05-01 20:01:39 on dist-test-slave-tzct"
server_key: "a1dc32e0d413270770d775685087b4ac"
server_key_iv: "3ddb0fbff15270d8b8641f5f585217fa"
server_key_version: "encryptionkey@0"
I20240501 20:01:39.746959 28231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:39.766410 28231 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:39.767061 28231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:39.767207 28231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:39.767724 28231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:39.768597 28231 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:39.768642 28231 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:39.768685 28231 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:39.768713 28231 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:39.776698 28231 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.66:35453
I20240501 20:01:39.776757 28358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.66:35453 every 8 connection(s)
I20240501 20:01:39.777626 28231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/data/info.pb
I20240501 20:01:39.785056 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.779262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.66:38919 (local address 127.8.168.126:33789)
0501 20:01:39.779411 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:39.779415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:39.780269 (+ 854us) server_negotiation.cc:407] Connection header received
0501 20:01:39.781105 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:39.781109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:39.781167 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:39.781239 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:39.782567 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.783055 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.783648 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.783824 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.784565 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:39.784586 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:39.784588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:39.784600 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:39.784618 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:39.784625 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:39.784685 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:39.784850 (+ 165us) server_negotiation.cc:299] Negotiation successful
0501 20:01:39.784910 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240501 20:01:39.785686 28361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.779539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33789 (local address 127.8.168.66:38919)
0501 20:01:39.780137 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:39.780169 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:39.780898 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:39.781402 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:39.781410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:39.781486 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:39.782424 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:39.782438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.783166 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.783169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:39.783540 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:39.783547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.783751 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.784347 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 20:01:39.784348 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:39.784382 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 20:01:39.784435 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:39.784698 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:39.784706 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:39.784711 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 20:01:39.784843 (+ 132us) client_negotiation.cc:240] Negotiation successful
0501 20:01:39.785072 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":93,"threads_started":1}
I20240501 20:01:39.786139 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 28231
I20240501 20:01:39.786269 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/wal/instance
I20240501 20:01:39.786573 8865 external_mini_cluster.cc:1428] Setting key 8bf618cafe390d2d5afd5f427aad9e86
I20240501 20:01:39.786767 28359 heartbeater.cc:347] Connected to a master server at 127.8.168.126:33789
I20240501 20:01:39.787039 28359 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:39.787593 28359 heartbeater.cc:511] Master 127.8.168.126:33789 requested a full tablet report, sending...
I20240501 20:01:39.788534 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.168.67:0
--local_ip_for_outbound_sockets=127.8.168.67
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33789
--builtin_ntp_servers=127.8.168.84:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240501 20:01:39.788628 28040 ts_manager.cc:194] Registered new tserver with Master: 3e7a7fcfab5e482588f4aff3e485175c (127.8.168.66:35453)
I20240501 20:01:39.789198 28040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.168.66:38919
W20240501 20:01:39.893990 28362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:39.894201 28362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:39.894249 28362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:39.897279 28362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:39.897329 28362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:39.897398 28362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.168.67
I20240501 20:01:39.901037 28362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.168.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.168.67
--webserver_port=0
--tserver_master_addrs=127.8.168.126:33789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.168.67
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:39.901808 28362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:39.902835 28362 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:39.910319 28367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:39.910624 28368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:39.911077 28370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:39.911522 28362 server_base.cc:1034] running on GCE node
I20240501 20:01:39.911850 28362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:39.912349 28362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:39.913486 28362 hybrid_clock.cc:648] HybridClock initialized: now 1714593699913462 us; error 39 us; skew 500 ppm
I20240501 20:01:39.915239 28362 webserver.cc:460] Webserver started at http://127.8.168.67:34949/ using document root <none> and password file <none>
I20240501 20:01:39.915731 28362 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:39.915781 28362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:39.915927 28362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:39.917620 28362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/data/instance:
uuid: "1a297a3cd3e74ed599731ad9adf66703"
format_stamp: "Formatted at 2024-05-01 20:01:39 on dist-test-slave-tzct"
server_key: "c3d4e0bb36e2a74a45b4b22b0026c5d3"
server_key_iv: "c0b1c28a3d5d7fa27bc09f4e48886999"
server_key_version: "encryptionkey@0"
I20240501 20:01:39.918043 28362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance:
uuid: "1a297a3cd3e74ed599731ad9adf66703"
format_stamp: "Formatted at 2024-05-01 20:01:39 on dist-test-slave-tzct"
server_key: "c3d4e0bb36e2a74a45b4b22b0026c5d3"
server_key_iv: "c0b1c28a3d5d7fa27bc09f4e48886999"
server_key_version: "encryptionkey@0"
I20240501 20:01:39.921085 28362 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:01:39.923079 28376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:39.924082 28362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:39.924168 28362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/wal
uuid: "1a297a3cd3e74ed599731ad9adf66703"
format_stamp: "Formatted at 2024-05-01 20:01:39 on dist-test-slave-tzct"
server_key: "c3d4e0bb36e2a74a45b4b22b0026c5d3"
server_key_iv: "c0b1c28a3d5d7fa27bc09f4e48886999"
server_key_version: "encryptionkey@0"
I20240501 20:01:39.924257 28362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:39.943650 28362 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:39.944322 28362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:39.944465 28362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:39.945025 28362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:39.945855 28362 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:39.945904 28362 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:39.945946 28362 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:39.945974 28362 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:39.954433 28362 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.67:42715
I20240501 20:01:39.954465 28489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.67:42715 every 8 connection(s)
I20240501 20:01:39.955323 28362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/data/info.pb
I20240501 20:01:39.962644 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.956955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:32797 (local address 127.8.168.126:33789)
0501 20:01:39.957098 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:39.957103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:39.958015 (+ 912us) server_negotiation.cc:407] Connection header received
0501 20:01:39.958748 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:39.958751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:39.958798 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:39.958862 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:39.960178 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.960676 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.961283 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.961433 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.962171 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:39.962189 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:39.962192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:39.962203 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:39.962224 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:39.962231 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:39.962290 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:39.962478 (+ 188us) server_negotiation.cc:299] Negotiation successful
0501 20:01:39.962529 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240501 20:01:39.963218 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.957307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:33789 (local address 127.8.168.67:32797)
0501 20:01:39.957889 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:39.957921 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:39.958558 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:39.958996 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:39.959004 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:39.959077 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:01:39.960027 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:39.960059 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.960795 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.960798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:39.961169 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:39.961177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.961342 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.961951 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0501 20:01:39.961953 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:01:39.961986 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0501 20:01:39.962044 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:01:39.962321 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:01:39.962327 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:01:39.962332 (+ 5us) client_negotiation.cc:769] Sending connection context
0501 20:01:39.962465 (+ 133us) client_negotiation.cc:240] Negotiation successful
0501 20:01:39.962645 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":99,"threads_started":1}
I20240501 20:01:39.964345 28490 heartbeater.cc:347] Connected to a master server at 127.8.168.126:33789
I20240501 20:01:39.964560 28490 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:39.964699 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 28362
I20240501 20:01:39.964864 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/wal/instance
I20240501 20:01:39.965097 8865 external_mini_cluster.cc:1428] Setting key e9feca911cc88d606f9e98012a0ceff9
I20240501 20:01:39.965132 28490 heartbeater.cc:511] Master 127.8.168.126:33789 requested a full tablet report, sending...
I20240501 20:01:39.966220 28040 ts_manager.cc:194] Registered new tserver with Master: 1a297a3cd3e74ed599731ad9adf66703 (127.8.168.67:42715)
I20240501 20:01:39.966840 28040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.168.67:32797
I20240501 20:01:39.967010 8865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 20:01:39.971510 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.968713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53926 (local address 127.8.168.126:33789)
0501 20:01:39.968852 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:39.968854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:39.968936 (+ 82us) server_negotiation.cc:407] Connection header received
0501 20:01:39.969065 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:39.969067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:39.969105 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:39.969169 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:39.969503 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.969995 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.970584 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.970708 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.971199 (+ 491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:39.971213 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:39.971215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:39.971225 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:39.971242 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:39.971249 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:39.971304 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:39.971370 (+ 66us) server_negotiation.cc:299] Negotiation successful
0501 20:01:39.971408 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240501 20:01:39.973953 28040 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:01:39.976317 28040 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:01:39.988469 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.983790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:39483 (local address 127.0.0.1:53356)
0501 20:01:39.984580 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:39.984618 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:39.984768 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:39.985673 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:39.985675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:39.985689 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:39.985962 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:39.985971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.987133 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.987136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:39.987970 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:39.987979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.988079 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.988135 (+ 56us) client_negotiation.cc:769] Sending connection context
0501 20:01:39.988226 (+ 91us) client_negotiation.cc:240] Negotiation successful
0501 20:01:39.988310 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":100,"threads_started":1}
I20240501 20:01:39.988829 28501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.984130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.67:42715 (local address 127.0.0.1:55416)
0501 20:01:39.985029 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:39.985040 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:39.985114 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:39.985611 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:39.985617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:39.985639 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:39.985962 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:39.985971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.987466 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.987470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:39.988438 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:39.988448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.988548 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.988565 (+ 17us) client_negotiation.cc:769] Sending connection context
0501 20:01:39.988612 (+ 47us) client_negotiation.cc:240] Negotiation successful
0501 20:01:39.988664 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":85,"threads_started":1}
I20240501 20:01:39.988845 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.983879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:35453 (local address 127.0.0.1:54698)
0501 20:01:39.984859 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:39.984872 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:39.984949 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:39.985855 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:39.985857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:39.985872 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:39.986058 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:39.986065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.987466 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.987470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:39.988438 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:39.988448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.988549 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.988565 (+ 16us) client_negotiation.cc:769] Sending connection context
0501 20:01:39.988612 (+ 47us) client_negotiation.cc:240] Negotiation successful
0501 20:01:39.988664 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":109,"threads_started":1}
I20240501 20:01:39.989602 28503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.983951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.8.168.65:39483)
0501 20:01:39.985297 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:39.985303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:39.985327 (+ 24us) server_negotiation.cc:407] Connection header received
0501 20:01:39.985401 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:39.985405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:39.985528 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:39.986116 (+ 588us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:39.986241 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.987027 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.988083 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.989253 (+ 1170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.989377 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:39.989434 (+ 57us) server_negotiation.cc:299] Negotiation successful
0501 20:01:39.989500 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1245,"thread_start_us":109,"threads_started":1}
I20240501 20:01:39.989660 28505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.984571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54698 (local address 127.8.168.66:35453)
0501 20:01:39.985350 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:39.985356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:39.985376 (+ 20us) server_negotiation.cc:407] Connection header received
0501 20:01:39.985470 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:39.985474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:39.985615 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:39.985740 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:39.986188 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.987337 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.988740 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.989247 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.989387 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:39.989463 (+ 76us) server_negotiation.cc:299] Negotiation successful
0501 20:01:39.989536 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":103,"threads_started":1}
I20240501 20:01:39.989967 28506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:39.984554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.8.168.67:42715)
0501 20:01:39.985116 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:39.985123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:39.985148 (+ 25us) server_negotiation.cc:407] Connection header received
0501 20:01:39.985258 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:39.985264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:39.985447 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:39.985591 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:39.986089 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.987344 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:39.989094 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:39.989576 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:39.989692 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:39.989755 (+ 63us) server_negotiation.cc:299] Negotiation successful
0501 20:01:39.989829 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":133,"threads_started":1}
I20240501 20:01:39.991910 28162 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc23907af3c41a38947266e3154e1eb (DEFAULT_TABLE table=test-table [id=298500680f7a4899922681c9280aedfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:39.991926 28293 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc23907af3c41a38947266e3154e1eb (DEFAULT_TABLE table=test-table [id=298500680f7a4899922681c9280aedfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:39.991927 28424 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc23907af3c41a38947266e3154e1eb (DEFAULT_TABLE table=test-table [id=298500680f7a4899922681c9280aedfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:01:39.992770 28162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc23907af3c41a38947266e3154e1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:39.992775 28424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc23907af3c41a38947266e3154e1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:39.992775 28293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc23907af3c41a38947266e3154e1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:39.998205 28507 tablet_bootstrap.cc:492] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20: Bootstrap starting.
I20240501 20:01:39.999408 28508 tablet_bootstrap.cc:492] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c: Bootstrap starting.
I20240501 20:01:39.999400 28509 tablet_bootstrap.cc:492] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703: Bootstrap starting.
I20240501 20:01:40.001224 28507 tablet_bootstrap.cc:654] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:40.001367 28509 tablet_bootstrap.cc:654] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:40.001367 28508 tablet_bootstrap.cc:654] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:40.001904 28507 log.cc:824] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:40.002096 28509 log.cc:824] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:40.002096 28508 log.cc:824] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:40.003599 28509 tablet_bootstrap.cc:492] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703: No bootstrap required, opened a new log
I20240501 20:01:40.003670 28508 tablet_bootstrap.cc:492] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c: No bootstrap required, opened a new log
I20240501 20:01:40.003670 28507 tablet_bootstrap.cc:492] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20: No bootstrap required, opened a new log
I20240501 20:01:40.003772 28509 ts_tablet_manager.cc:1367] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:01:40.003834 28507 ts_tablet_manager.cc:1367] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:01:40.003834 28508 ts_tablet_manager.cc:1367] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:01:40.007381 28509 raft_consensus.cc:348] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e090c185833a4b938dbda0a99700cc20" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39483 } } peers { permanent_uuid: "3e7a7fcfab5e482588f4aff3e485175c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35453 } } peers { permanent_uuid: "1a297a3cd3e74ed599731ad9adf66703" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 42715 } }
I20240501 20:01:40.007416 28508 raft_consensus.cc:348] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e090c185833a4b938dbda0a99700cc20" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39483 } } peers { permanent_uuid: "3e7a7fcfab5e482588f4aff3e485175c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35453 } } peers { permanent_uuid: "1a297a3cd3e74ed599731ad9adf66703" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 42715 } }
I20240501 20:01:40.007409 28507 raft_consensus.cc:348] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e090c185833a4b938dbda0a99700cc20" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39483 } } peers { permanent_uuid: "3e7a7fcfab5e482588f4aff3e485175c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35453 } } peers { permanent_uuid: "1a297a3cd3e74ed599731ad9adf66703" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 42715 } }
I20240501 20:01:40.007558 28509 raft_consensus.cc:374] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:40.007588 28508 raft_consensus.cc:374] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:40.007588 28507 raft_consensus.cc:374] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:40.007612 28509 raft_consensus.cc:729] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a297a3cd3e74ed599731ad9adf66703, State: Initialized, Role: FOLLOWER
I20240501 20:01:40.007634 28507 raft_consensus.cc:729] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e090c185833a4b938dbda0a99700cc20, State: Initialized, Role: FOLLOWER
I20240501 20:01:40.007634 28508 raft_consensus.cc:729] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e7a7fcfab5e482588f4aff3e485175c, State: Initialized, Role: FOLLOWER
I20240501 20:01:40.007999 28509 consensus_queue.cc:260] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e090c185833a4b938dbda0a99700cc20" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39483 } } peers { permanent_uuid: "3e7a7fcfab5e482588f4aff3e485175c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35453 } } peers { permanent_uuid: "1a297a3cd3e74ed599731ad9adf66703" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 42715 } }
I20240501 20:01:40.008059 28508 consensus_queue.cc:260] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e090c185833a4b938dbda0a99700cc20" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39483 } } peers { permanent_uuid: "3e7a7fcfab5e482588f4aff3e485175c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35453 } } peers { permanent_uuid: "1a297a3cd3e74ed599731ad9adf66703" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 42715 } }
I20240501 20:01:40.008059 28507 consensus_queue.cc:260] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e090c185833a4b938dbda0a99700cc20" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39483 } } peers { permanent_uuid: "3e7a7fcfab5e482588f4aff3e485175c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35453 } } peers { permanent_uuid: "1a297a3cd3e74ed599731ad9adf66703" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 42715 } }
I20240501 20:01:40.008621 28490 heartbeater.cc:503] Master 127.8.168.126:33789 was elected leader, sending a full tablet report...
I20240501 20:01:40.008865 28508 ts_tablet_manager.cc:1398] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240501 20:01:40.008870 28509 ts_tablet_manager.cc:1398] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 20:01:40.008865 28507 ts_tablet_manager.cc:1398] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20240501 20:01:40.009191 28359 heartbeater.cc:503] Master 127.8.168.126:33789 was elected leader, sending a full tablet report...
I20240501 20:01:40.009191 28228 heartbeater.cc:503] Master 127.8.168.126:33789 was elected leader, sending a full tablet report...
W20240501 20:01:40.028438 28360 tablet.cc:2369] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:01:40.107471 28229 tablet.cc:2369] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:01:40.170713 28513 raft_consensus.cc:482] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:40.170903 28513 raft_consensus.cc:504] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e090c185833a4b938dbda0a99700cc20" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39483 } } peers { permanent_uuid: "3e7a7fcfab5e482588f4aff3e485175c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35453 } } peers { permanent_uuid: "1a297a3cd3e74ed599731ad9adf66703" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 42715 } }
I20240501 20:01:40.171977 28513 leader_election.cc:288] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e090c185833a4b938dbda0a99700cc20 (127.8.168.65:39483), 3e7a7fcfab5e482588f4aff3e485175c (127.8.168.66:35453)
I20240501 20:01:40.174818 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:40.172187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.66:35453 (local address 127.8.168.67:55301)
0501 20:01:40.172307 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:40.172322 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:40.172427 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:40.172668 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:40.172670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:40.172686 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:40.172952 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:40.172960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:40.173816 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:40.173819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:40.174511 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:40.174520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:40.174624 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:40.174641 (+ 17us) client_negotiation.cc:769] Sending connection context
0501 20:01:40.174674 (+ 33us) client_negotiation.cc:240] Negotiation successful
0501 20:01:40.174717 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20240501 20:01:40.175354 28505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:40.172317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:55301 (local address 127.8.168.66:35453)
0501 20:01:40.172428 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:40.172432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:40.172449 (+ 17us) server_negotiation.cc:407] Connection header received
0501 20:01:40.172509 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:40.172512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:40.172568 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:40.172663 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:40.173068 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:40.173715 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:40.174646 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:40.175106 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:40.175142 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:40.175189 (+ 47us) server_negotiation.cc:299] Negotiation successful
0501 20:01:40.175239 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240501 20:01:40.175783 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:40.172187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:39483 (local address 127.8.168.67:43429)
0501 20:01:40.172652 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:40.172664 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 20:01:40.172784 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:01:40.173035 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:01:40.173038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:01:40.173053 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0501 20:01:40.173346 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:40.173353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:40.174157 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:01:40.174160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:01:40.174834 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:01:40.174843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:01:40.175543 (+ 700us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:40.175556 (+ 13us) client_negotiation.cc:769] Sending connection context
0501 20:01:40.175617 (+ 61us) client_negotiation.cc:240] Negotiation successful
0501 20:01:40.175657 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":126,"thread_start_us":136,"threads_started":1}
I20240501 20:01:40.175839 28503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:40.172301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.168.67:43429 (local address 127.8.168.65:39483)
0501 20:01:40.172441 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:40.172446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:40.172759 (+ 313us) server_negotiation.cc:407] Connection header received
0501 20:01:40.172883 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:40.172887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:40.172939 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:40.173024 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 20:01:40.173463 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:40.174034 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:40.174977 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:40.175431 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:40.175467 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:40.175675 (+ 208us) server_negotiation.cc:299] Negotiation successful
0501 20:01:40.175746 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20240501 20:01:40.176004 28313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc23907af3c41a38947266e3154e1eb" candidate_uuid: "1a297a3cd3e74ed599731ad9adf66703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e7a7fcfab5e482588f4aff3e485175c" is_pre_election: true
I20240501 20:01:40.176288 28313 raft_consensus.cc:2454] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a297a3cd3e74ed599731ad9adf66703 in term 0.
I20240501 20:01:40.176479 28182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc23907af3c41a38947266e3154e1eb" candidate_uuid: "1a297a3cd3e74ed599731ad9adf66703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e090c185833a4b938dbda0a99700cc20" is_pre_election: true
I20240501 20:01:40.176756 28182 raft_consensus.cc:2454] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a297a3cd3e74ed599731ad9adf66703 in term 0.
I20240501 20:01:40.176780 28377 leader_election.cc:302] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a297a3cd3e74ed599731ad9adf66703, 3e7a7fcfab5e482588f4aff3e485175c; no voters:
I20240501 20:01:40.177021 28513 raft_consensus.cc:2781] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:40.177101 28513 raft_consensus.cc:482] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:40.177151 28513 raft_consensus.cc:3037] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:40.178038 28513 raft_consensus.cc:504] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e090c185833a4b938dbda0a99700cc20" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39483 } } peers { permanent_uuid: "3e7a7fcfab5e482588f4aff3e485175c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35453 } } peers { permanent_uuid: "1a297a3cd3e74ed599731ad9adf66703" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 42715 } }
I20240501 20:01:40.178359 28513 leader_election.cc:288] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [CANDIDATE]: Term 1 election: Requested vote from peers e090c185833a4b938dbda0a99700cc20 (127.8.168.65:39483), 3e7a7fcfab5e482588f4aff3e485175c (127.8.168.66:35453)
I20240501 20:01:40.178649 28182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc23907af3c41a38947266e3154e1eb" candidate_uuid: "1a297a3cd3e74ed599731ad9adf66703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e090c185833a4b938dbda0a99700cc20"
I20240501 20:01:40.178678 28313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc23907af3c41a38947266e3154e1eb" candidate_uuid: "1a297a3cd3e74ed599731ad9adf66703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e7a7fcfab5e482588f4aff3e485175c"
I20240501 20:01:40.178762 28182 raft_consensus.cc:3037] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:40.178786 28313 raft_consensus.cc:3037] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:40.179622 28182 raft_consensus.cc:2454] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a297a3cd3e74ed599731ad9adf66703 in term 1.
I20240501 20:01:40.179695 28313 raft_consensus.cc:2454] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a297a3cd3e74ed599731ad9adf66703 in term 1.
I20240501 20:01:40.179903 28378 leader_election.cc:302] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a297a3cd3e74ed599731ad9adf66703, e090c185833a4b938dbda0a99700cc20; no voters:
I20240501 20:01:40.180061 28513 raft_consensus.cc:2781] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:40.180250 28513 raft_consensus.cc:686] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [term 1 LEADER]: Becoming Leader. State: Replica: 1a297a3cd3e74ed599731ad9adf66703, State: Running, Role: LEADER
I20240501 20:01:40.180534 28513 consensus_queue.cc:237] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e090c185833a4b938dbda0a99700cc20" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39483 } } peers { permanent_uuid: "3e7a7fcfab5e482588f4aff3e485175c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35453 } } peers { permanent_uuid: "1a297a3cd3e74ed599731ad9adf66703" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 42715 } }
I20240501 20:01:40.183549 28039 catalog_manager.cc:5484] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a297a3cd3e74ed599731ad9adf66703 (127.8.168.67). New cstate: current_term: 1 leader_uuid: "1a297a3cd3e74ed599731ad9adf66703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e090c185833a4b938dbda0a99700cc20" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e7a7fcfab5e482588f4aff3e485175c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a297a3cd3e74ed599731ad9adf66703" member_type: VOTER last_known_addr { host: "127.8.168.67" port: 42715 } health_report { overall_health: HEALTHY } } }
W20240501 20:01:40.206203 28491 tablet.cc:2369] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:01:40.222602 28506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:40.219433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.8.168.67:42715)
0501 20:01:40.219544 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:40.219548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:40.219582 (+ 34us) server_negotiation.cc:407] Connection header received
0501 20:01:40.219737 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:40.219739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:40.219797 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:40.219925 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:01:40.220321 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:40.220889 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:40.221558 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:40.221706 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:40.221789 (+ 83us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:01:40.222235 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:01:40.222333 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:40.222452 (+ 119us) server_negotiation.cc:299] Negotiation successful
0501 20:01:40.222496 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240501 20:01:40.227818 28182 raft_consensus.cc:1261] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20 [term 1 FOLLOWER]: Refusing update from remote peer 1a297a3cd3e74ed599731ad9adf66703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:40.227855 28313 raft_consensus.cc:1261] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c [term 1 FOLLOWER]: Refusing update from remote peer 1a297a3cd3e74ed599731ad9adf66703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:40.228448 28513 consensus_queue.cc:1035] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e090c185833a4b938dbda0a99700cc20" member_type: VOTER last_known_addr { host: "127.8.168.65" port: 39483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:40.228618 28517 consensus_queue.cc:1035] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e7a7fcfab5e482588f4aff3e485175c" member_type: VOTER last_known_addr { host: "127.8.168.66" port: 35453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:40.240662 28521 mvcc.cc:205] Tried to move back new op lower bound from 7022975796126031872 to 7022975795942014976. Current Snapshot: MvccSnapshot[applied={T|T < 7022975796126031872}]
I20240501 20:01:40.246649 28039 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53926:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:01:40.246793 28039 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53926:
table { table_name: "test-table" } modify_external_catalogs: true
I20240501 20:01:40.248966 28039 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d537bc35eac543d4afb06ba874bf5361: Sending DeleteTablet for 3 replicas of tablet 9fc23907af3c41a38947266e3154e1eb
I20240501 20:01:40.249729 28293 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fc23907af3c41a38947266e3154e1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:40 UTC) from {username='slave'} at 127.0.0.1:54698
I20240501 20:01:40.249843 28162 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fc23907af3c41a38947266e3154e1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:40 UTC) from {username='slave'} at 127.0.0.1:53356
I20240501 20:01:40.250195 28530 tablet_replica.cc:330] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c: stopping tablet replica
I20240501 20:01:40.250236 28531 tablet_replica.cc:330] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20: stopping tablet replica
I20240501 20:01:40.250245 28424 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fc23907af3c41a38947266e3154e1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:01:40 UTC) from {username='slave'} at 127.0.0.1:55416
I20240501 20:01:40.250561 28531 raft_consensus.cc:2229] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:40.250568 28530 raft_consensus.cc:2229] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:40.250888 28531 raft_consensus.cc:2258] T 9fc23907af3c41a38947266e3154e1eb P e090c185833a4b938dbda0a99700cc20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:40.251034 28530 raft_consensus.cc:2258] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:40.251216 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 28100
I20240501 20:01:40.252476 28530 ts_tablet_manager.cc:1875] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:01:40.253629 28532 tablet_replica.cc:330] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703: stopping tablet replica
I20240501 20:01:40.253929 28532 raft_consensus.cc:2229] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:40.254345 28532 raft_consensus.cc:2258] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:40.255352 28530 ts_tablet_manager.cc:1888] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:01:40.255445 28530 log.cc:1196] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-1/wal/wals/9fc23907af3c41a38947266e3154e1eb
I20240501 20:01:40.255620 28532 ts_tablet_manager.cc:1875] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:01:40.255762 28530 ts_tablet_manager.cc:1909] T 9fc23907af3c41a38947266e3154e1eb P 3e7a7fcfab5e482588f4aff3e485175c: Deleting consensus metadata
I20240501 20:01:40.256979 28024 catalog_manager.cc:4830] TS 3e7a7fcfab5e482588f4aff3e485175c (127.8.168.66:35453): tablet 9fc23907af3c41a38947266e3154e1eb (table test-table [id=298500680f7a4899922681c9280aedfd]) successfully deleted
I20240501 20:01:40.258178 28532 ts_tablet_manager.cc:1888] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 20:01:40.258296 28532 log.cc:1196] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703: Deleting WAL directory at /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1714593353892612-8865-0/minicluster-data/ts-2/wal/wals/9fc23907af3c41a38947266e3154e1eb
I20240501 20:01:40.258618 28532 ts_tablet_manager.cc:1909] T 9fc23907af3c41a38947266e3154e1eb P 1a297a3cd3e74ed599731ad9adf66703: Deleting consensus metadata
W20240501 20:01:40.259099 28025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.168.65:39483 (error 108)
I20240501 20:01:40.259661 28026 catalog_manager.cc:4830] TS 1a297a3cd3e74ed599731ad9adf66703 (127.8.168.67:42715): tablet 9fc23907af3c41a38947266e3154e1eb (table test-table [id=298500680f7a4899922681c9280aedfd]) successfully deleted
W20240501 20:01:40.260119 28502 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:01:40.259833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.65:39483 (local address 127.0.0.1:53368)
0501 20:01:40.259951 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0501 20:01:40.260017 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.65:39483: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":27}
I20240501 20:01:40.260164 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 28231
W20240501 20:01:40.261075 28025 catalog_manager.cc:4573] TS e090c185833a4b938dbda0a99700cc20 (127.8.168.65:39483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fc23907af3c41a38947266e3154e1eb: Network error: Client connection negotiation failed: client connection to 127.8.168.65:39483: connect: Connection reset by peer (error 104)
I20240501 20:01:40.265113 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 28362
I20240501 20:01:40.269771 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 28009
2024-05-01T20:01:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1073 ms)
[----------] 6 tests from EncryptionPolicyTest (10188 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:40 dist-test-slave-tzct krb5kdc[28536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:40 dist-test-slave-tzct krb5kdc[28536](info): set up 2 sockets
May 01 20:01:40 dist-test-slave-tzct krb5kdc[28536](info): commencing operation
krb5kdc: starting...
W20240501 20:01:41.169322 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.880s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:41 dist-test-slave-tzct krb5kdc[28536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 20:01:41.287688 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37399
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:41.394645 28549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:41.394896 28549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:41.394973 28549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:41.398072 28549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:41.398144 28549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:41.398175 28549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:41.398203 28549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:41.398234 28549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:41.402383 28549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37399
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:41.403270 28549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:41.404098 28549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:41.411329 28555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:41.411692 28554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:41.412387 28557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:41.412959 28549 server_base.cc:1034] running on GCE node
I20240501 20:01:41.413445 28549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:41.414364 28549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:41.415517 28549 hybrid_clock.cc:648] HybridClock initialized: now 1714593701415515 us; error 40 us; skew 500 ppm
May 01 20:01:41 dist-test-slave-tzct krb5kdc[28536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:41.418188 28549 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 20:01:41.419142 28549 webserver.cc:460] Webserver started at http://127.0.0.1:39307/ using document root <none> and password file <none>
I20240501 20:01:41.419672 28549 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:41.419729 28549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:41.419906 28549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:41.421744 28549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "2602886f9a514c6aa6770ff62284d2a1"
format_stamp: "Formatted at 2024-05-01 20:01:41 on dist-test-slave-tzct"
server_key: "25f35c62eaf503cf2887c1f4a14f6cb7"
server_key_iv: "c430de0a8030663dd0d0ea58bca87306"
server_key_version: "encryptionkey@0"
I20240501 20:01:41.422224 28549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "2602886f9a514c6aa6770ff62284d2a1"
format_stamp: "Formatted at 2024-05-01 20:01:41 on dist-test-slave-tzct"
server_key: "25f35c62eaf503cf2887c1f4a14f6cb7"
server_key_iv: "c430de0a8030663dd0d0ea58bca87306"
server_key_version: "encryptionkey@0"
I20240501 20:01:41.425587 28549 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:01:41.427788 28564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:41.428876 28549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:41.429005 28549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "2602886f9a514c6aa6770ff62284d2a1"
format_stamp: "Formatted at 2024-05-01 20:01:41 on dist-test-slave-tzct"
server_key: "25f35c62eaf503cf2887c1f4a14f6cb7"
server_key_iv: "c430de0a8030663dd0d0ea58bca87306"
server_key_version: "encryptionkey@0"
I20240501 20:01:41.429102 28549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:41.444672 28549 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:41.447520 28549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:41.447693 28549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:41.454370 28549 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37399
I20240501 20:01:41.454612 28616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37399 every 8 connection(s)
I20240501 20:01:41.455439 28549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:41.458084 28617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:41.463129 28617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1: Bootstrap starting.
I20240501 20:01:41.463470 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 28549
I20240501 20:01:41.463575 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:41.463793 8865 external_mini_cluster.cc:1428] Setting key 0fd97648c0df29e502adebde8b65469d
I20240501 20:01:41.465571 28617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:41.466423 28617 log.cc:824] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:41.468189 28617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1: No bootstrap required, opened a new log
May 01 20:01:41 dist-test-slave-tzct krb5kdc[28536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240501 20:01:41.471503 28617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2602886f9a514c6aa6770ff62284d2a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37399 } }
I20240501 20:01:41.471736 28617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:41.471796 28617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2602886f9a514c6aa6770ff62284d2a1, State: Initialized, Role: FOLLOWER
I20240501 20:01:41.472215 28617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2602886f9a514c6aa6770ff62284d2a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37399 } }
I20240501 20:01:41.472332 28617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:41.472404 28617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:41.472512 28617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:41.473726 28617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2602886f9a514c6aa6770ff62284d2a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37399 } }
I20240501 20:01:41.474159 28617 leader_election.cc:302] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2602886f9a514c6aa6770ff62284d2a1; no voters:
I20240501 20:01:41.474579 28617 leader_election.cc:288] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:41.474715 28622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:41.475008 28622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [term 1 LEADER]: Becoming Leader. State: Replica: 2602886f9a514c6aa6770ff62284d2a1, State: Running, Role: LEADER
I20240501 20:01:41.475461 28622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2602886f9a514c6aa6770ff62284d2a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37399 } }
I20240501 20:01:41.476147 28617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:41.477223 28624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2602886f9a514c6aa6770ff62284d2a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2602886f9a514c6aa6770ff62284d2a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37399 } } }
I20240501 20:01:41.477346 28624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:41.477663 28623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2602886f9a514c6aa6770ff62284d2a1. Latest consensus state: current_term: 1 leader_uuid: "2602886f9a514c6aa6770ff62284d2a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2602886f9a514c6aa6770ff62284d2a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37399 } } }
I20240501 20:01:41.477761 28623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:41.481125 28620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:41.465015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40737 (local address 127.0.0.1:37399)
0501 20:01:41.465462 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:41.465473 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:41.465513 (+ 40us) server_negotiation.cc:407] Connection header received
0501 20:01:41.466281 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:41.466307 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:41.466690 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:41.467072 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:41.467975 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:41.468820 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:41.469472 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:41.469736 (+ 264us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:41.472246 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:41.472274 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:41.472287 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:41.472328 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:41.474898 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:41.477739 (+ 2841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:41.477745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:41.477751 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:41.477864 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:41.478248 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:41.478251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:41.478253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:41.479057 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:41.479890 (+ 833us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:41.480085 (+ 195us) server_negotiation.cc:299] Negotiation successful
0501 20:01:41.480314 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":118,"threads_started":1}
W20240501 20:01:41.482954 28637 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:01:41.483031 28637 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:01:41.483093 28631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:41.484136 28631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:41.484418 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:01:41.494014 28631 catalog_manager.cc:1347] Generated new cluster ID: 28194fbbd6034bc28fef9c7b1de199ba
I20240501 20:01:41.494107 28631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:41.507975 28620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:41.489135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45019 (local address 127.0.0.1:37399)
0501 20:01:41.489239 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:41.489243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:41.492215 (+ 2972us) server_negotiation.cc:407] Connection header received
0501 20:01:41.492269 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:41.492272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:41.492330 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:41.492406 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:41.493564 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:41.501289 (+ 7725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:41.502580 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:41.502792 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:41.504005 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:41.504029 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:41.504032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:41.504061 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:41.505944 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:41.506489 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:41.506492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:41.506494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:41.506546 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:41.506811 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:41.506814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:41.506816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:41.506985 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:41.507113 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:41.507702 (+ 589us) server_negotiation.cc:299] Negotiation successful
0501 20:01:41.507793 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240501 20:01:41.515038 28631 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:41.516439 28631 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:41.526103 28631 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2602886f9a514c6aa6770ff62284d2a1: Generated new TSK 0
I20240501 20:01:41.527102 28631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:41.532332 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 28549
W20240501 20:01:41.536896 28534 connection.cc:535] client connection to 127.0.0.1:37399 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37399): Connection reset by peer (error 104)
2024-05-01T20:01:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1261 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:41 dist-test-slave-tzct krb5kdc[28648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:41 dist-test-slave-tzct krb5kdc[28648](info): set up 2 sockets
May 01 20:01:41 dist-test-slave-tzct krb5kdc[28648](info): commencing operation
krb5kdc: starting...
W20240501 20:01:42.425010 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.874s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:42 dist-test-slave-tzct krb5kdc[28648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 20:01:42.537933 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41603
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:42.643297 28661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:42.643529 28661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:42.643582 28661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:42.646605 28661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:42.646662 28661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:42.646684 28661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:42.646703 28661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:42.646725 28661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:42.650796 28661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41603
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:42.651607 28661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:42.652379 28661 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:42.659482 28669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:42.659605 28666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:42.660056 28667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:42.660519 28661 server_base.cc:1034] running on GCE node
I20240501 20:01:42.661026 28661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:42.661891 28661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:42.663044 28661 hybrid_clock.cc:648] HybridClock initialized: now 1714593702663032 us; error 26 us; skew 500 ppm
May 01 20:01:42 dist-test-slave-tzct krb5kdc[28648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:42.665704 28661 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 20:01:42.666599 28661 webserver.cc:460] Webserver started at http://127.0.0.1:33177/ using document root <none> and password file <none>
I20240501 20:01:42.667111 28661 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:42.667166 28661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:42.667331 28661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:42.669097 28661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "656159840ae84654aef2dd61ec97f13f"
format_stamp: "Formatted at 2024-05-01 20:01:42 on dist-test-slave-tzct"
server_key: "b83624b4519dadae77d5c1642bd530f3"
server_key_iv: "890c2765d54111e29cbb3b917cb6f258"
server_key_version: "encryptionkey@0"
I20240501 20:01:42.669543 28661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "656159840ae84654aef2dd61ec97f13f"
format_stamp: "Formatted at 2024-05-01 20:01:42 on dist-test-slave-tzct"
server_key: "b83624b4519dadae77d5c1642bd530f3"
server_key_iv: "890c2765d54111e29cbb3b917cb6f258"
server_key_version: "encryptionkey@0"
I20240501 20:01:42.672678 28661 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:01:42.674795 28676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:42.675760 28661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 20:01:42.675876 28661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "656159840ae84654aef2dd61ec97f13f"
format_stamp: "Formatted at 2024-05-01 20:01:42 on dist-test-slave-tzct"
server_key: "b83624b4519dadae77d5c1642bd530f3"
server_key_iv: "890c2765d54111e29cbb3b917cb6f258"
server_key_version: "encryptionkey@0"
I20240501 20:01:42.675987 28661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:42.711795 28661 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:42.714993 28661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:42.715158 28661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:42.721747 28661 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41603
I20240501 20:01:42.721964 28728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41603 every 8 connection(s)
I20240501 20:01:42.722698 28661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:42.722983 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 28661
I20240501 20:01:42.723091 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:42.723347 8865 external_mini_cluster.cc:1428] Setting key 921c0e9e7bb787845dffeb4e01ff1ad9
I20240501 20:01:42.725590 28729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:42.730881 28732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:42.724556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39115 (local address 127.0.0.1:41603)
0501 20:01:42.724972 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:42.724980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:42.725200 (+ 220us) server_negotiation.cc:407] Connection header received
0501 20:01:42.725887 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:42.725905 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:42.726179 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:42.726478 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:42.726977 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:42.727725 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:42.728562 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:42.728787 (+ 225us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:42.729449 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:42.729504 (+ 55us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:42.729514 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:42.729860 (+ 346us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:42.729892 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:42.729908 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:42.730006 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:42.730187 (+ 181us) server_negotiation.cc:299] Negotiation successful
0501 20:01:42.730293 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":161,"threads_started":1}
I20240501 20:01:42.730902 28729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f: Bootstrap starting.
I20240501 20:01:42.733062 28729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:42.733497 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:01:42.733863 28729 log.cc:824] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:42.735808 28729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f: No bootstrap required, opened a new log
I20240501 20:01:42.738884 28729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656159840ae84654aef2dd61ec97f13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } }
I20240501 20:01:42.739055 28732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:42.735538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40921 (local address 127.0.0.1:41603)
0501 20:01:42.735659 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:42.735663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:42.735778 (+ 115us) server_negotiation.cc:407] Connection header received
0501 20:01:42.735880 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:42.735883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:42.735935 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:42.736017 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:42.736353 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:42.737414 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:42.737984 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:42.738172 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:42.738640 (+ 468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:42.738657 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:42.738660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:42.738672 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:42.738701 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:42.738712 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:42.738798 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:42.738899 (+ 101us) server_negotiation.cc:299] Negotiation successful
0501 20:01:42.738936 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240501 20:01:42.739115 28729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:42.739163 28729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 656159840ae84654aef2dd61ec97f13f, State: Initialized, Role: FOLLOWER
I20240501 20:01:42.739594 28729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656159840ae84654aef2dd61ec97f13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } }
I20240501 20:01:42.739715 28729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:42.739769 28729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:42.739874 28729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:42.740648 28729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656159840ae84654aef2dd61ec97f13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } }
I20240501 20:01:42.740979 28729 leader_election.cc:302] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 656159840ae84654aef2dd61ec97f13f; no voters:
I20240501 20:01:42.741230 28729 leader_election.cc:288] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:42.741312 28739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:42.741461 28739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [term 1 LEADER]: Becoming Leader. State: Replica: 656159840ae84654aef2dd61ec97f13f, State: Running, Role: LEADER
I20240501 20:01:42.741726 28739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656159840ae84654aef2dd61ec97f13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } }
I20240501 20:01:42.742311 28729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:42.743242 28741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "656159840ae84654aef2dd61ec97f13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656159840ae84654aef2dd61ec97f13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } } }
I20240501 20:01:42.743372 28741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:42.743652 28740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 656159840ae84654aef2dd61ec97f13f. Latest consensus state: current_term: 1 leader_uuid: "656159840ae84654aef2dd61ec97f13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656159840ae84654aef2dd61ec97f13f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } } }
I20240501 20:01:42.743741 28740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:42.743862 28748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:42.746649 28748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:42.751175 28748 catalog_manager.cc:1347] Generated new cluster ID: 1e3d2bcc8f2c4bf480dd298e703f90fb
I20240501 20:01:42.751240 28748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:42.759855 28748 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:42.760743 28748 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:42.772763 28748 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 656159840ae84654aef2dd61ec97f13f: Generated new TSK 0
I20240501 20:01:42.773433 28748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:42.782061 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 28661
W20240501 20:01:42.786672 28646 connection.cc:535] client connection to 127.0.0.1:41603 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41603): Connection reset by peer (error 104)
2024-05-01T20:01:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1249 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:42 dist-test-slave-tzct krb5kdc[28760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:42 dist-test-slave-tzct krb5kdc[28760](info): set up 2 sockets
May 01 20:01:42 dist-test-slave-tzct krb5kdc[28760](info): commencing operation
krb5kdc: starting...
W20240501 20:01:43.673002 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.872s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:43 dist-test-slave-tzct krb5kdc[28760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 20:01:43.782042 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:43.884680 28773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:43.884967 28773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:43.885044 28773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:43.888069 28773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:43.888135 28773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:43.888164 28773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:43.888190 28773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:43.888223 28773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:43.892323 28773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41349
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:43.893247 28773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:43.894093 28773 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:43.901036 28779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:43.901350 28781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:43.901834 28778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:43.902246 28773 server_base.cc:1034] running on GCE node
I20240501 20:01:43.902748 28773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:43.903640 28773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:43.904773 28773 hybrid_clock.cc:648] HybridClock initialized: now 1714593703904760 us; error 29 us; skew 500 ppm
May 01 20:01:43 dist-test-slave-tzct krb5kdc[28760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:43.907377 28773 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 20:01:43.908284 28773 webserver.cc:460] Webserver started at http://127.0.0.1:35479/ using document root <none> and password file <none>
I20240501 20:01:43.908845 28773 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:43.908906 28773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:43.909082 28773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:43.910846 28773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "79c41f8e463a4441a02818ffb5990f08"
format_stamp: "Formatted at 2024-05-01 20:01:43 on dist-test-slave-tzct"
server_key: "1f9ac918949ec74b601ab68659f194c3"
server_key_iv: "358109ef2061eea4837ad1a0adacdecb"
server_key_version: "encryptionkey@0"
I20240501 20:01:43.911301 28773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "79c41f8e463a4441a02818ffb5990f08"
format_stamp: "Formatted at 2024-05-01 20:01:43 on dist-test-slave-tzct"
server_key: "1f9ac918949ec74b601ab68659f194c3"
server_key_iv: "358109ef2061eea4837ad1a0adacdecb"
server_key_version: "encryptionkey@0"
I20240501 20:01:43.914624 28773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 20:01:43.916802 28788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:43.917886 28773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:43.918018 28773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "79c41f8e463a4441a02818ffb5990f08"
format_stamp: "Formatted at 2024-05-01 20:01:43 on dist-test-slave-tzct"
server_key: "1f9ac918949ec74b601ab68659f194c3"
server_key_iv: "358109ef2061eea4837ad1a0adacdecb"
server_key_version: "encryptionkey@0"
I20240501 20:01:43.918121 28773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:43.922873 28773 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:43.926120 28773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:43.926280 28773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:43.932597 28773 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41349
I20240501 20:01:43.932828 28840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41349 every 8 connection(s)
I20240501 20:01:43.933590 28773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:43.935968 28841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:43.936838 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 28773
I20240501 20:01:43.936956 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:43.937189 8865 external_mini_cluster.cc:1428] Setting key 35b0e332beb4ed614a309cac73dbbee9
I20240501 20:01:43.941766 28841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08: Bootstrap starting.
I20240501 20:01:43.941890 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:43.938297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40739 (local address 127.0.0.1:41349)
0501 20:01:43.938693 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:43.938706 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:43.938748 (+ 42us) server_negotiation.cc:407] Connection header received
0501 20:01:43.939438 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:43.939462 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:43.939755 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:43.940132 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:43.940776 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:43.940781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:43.940793 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:43.940804 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:43.940846 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:43.940866 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:43.941026 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:43.941100 (+ 74us) server_negotiation.cc:299] Negotiation successful
0501 20:01:43.941217 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20240501 20:01:43.943912 28841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:43.944445 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:01:43.944682 28841 log.cc:824] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:43.946621 28841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08: No bootstrap required, opened a new log
I20240501 20:01:43.948000 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:43.946427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57789 (local address 127.0.0.1:41349)
0501 20:01:43.946533 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:43.946536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:43.946729 (+ 193us) server_negotiation.cc:407] Connection header received
0501 20:01:43.946910 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:43.946913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:43.946944 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:43.947002 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:43.947674 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:43.947676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:43.947678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:43.947687 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:43.947713 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:43.947722 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:43.947785 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:43.947867 (+ 82us) server_negotiation.cc:299] Negotiation successful
0501 20:01:43.947893 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240501 20:01:43.950030 28841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c41f8e463a4441a02818ffb5990f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } }
I20240501 20:01:43.950202 28841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:43.950271 28841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c41f8e463a4441a02818ffb5990f08, State: Initialized, Role: FOLLOWER
I20240501 20:01:43.950644 28841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c41f8e463a4441a02818ffb5990f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } }
I20240501 20:01:43.950759 28841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:43.950816 28841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:43.950927 28841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:43.951669 28841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c41f8e463a4441a02818ffb5990f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } }
I20240501 20:01:43.951972 28841 leader_election.cc:302] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79c41f8e463a4441a02818ffb5990f08; no voters:
I20240501 20:01:43.952224 28841 leader_election.cc:288] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:43.952339 28851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:43.952551 28851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [term 1 LEADER]: Becoming Leader. State: Replica: 79c41f8e463a4441a02818ffb5990f08, State: Running, Role: LEADER
I20240501 20:01:43.952868 28851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c41f8e463a4441a02818ffb5990f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } }
I20240501 20:01:43.953246 28841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:43.954300 28852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79c41f8e463a4441a02818ffb5990f08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c41f8e463a4441a02818ffb5990f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } } }
I20240501 20:01:43.954421 28852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:43.954782 28860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:43.954741 28853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79c41f8e463a4441a02818ffb5990f08. Latest consensus state: current_term: 1 leader_uuid: "79c41f8e463a4441a02818ffb5990f08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c41f8e463a4441a02818ffb5990f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41349 } } }
I20240501 20:01:43.954900 28853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:43.957690 28860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:43.962875 28860 catalog_manager.cc:1347] Generated new cluster ID: ca91ca27d335441ca0689d61a749c04e
I20240501 20:01:43.962950 28860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:43.972851 28860 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:43.973147 28860 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:43.982308 28860 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 79c41f8e463a4441a02818ffb5990f08: Generated new TSK 0
I20240501 20:01:43.983072 28860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:43.991549 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 28773
2024-05-01T20:01:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:44 dist-test-slave-tzct krb5kdc[28872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:44 dist-test-slave-tzct krb5kdc[28872](info): set up 2 sockets
May 01 20:01:44 dist-test-slave-tzct krb5kdc[28872](info): commencing operation
krb5kdc: starting...
W20240501 20:01:44.880965 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.871s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:44 dist-test-slave-tzct krb5kdc[28872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 20:01:44.993690 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:45.100272 28885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:45.100517 28885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:45.100576 28885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:45.103708 28885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:45.103778 28885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:45.103801 28885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:45.103817 28885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:45.103834 28885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:45.107991 28885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:45.108943 28885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:45.109755 28885 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:45.116652 28893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:45.116941 28890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:45.117452 28891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:45.117889 28885 server_base.cc:1034] running on GCE node
I20240501 20:01:45.118368 28885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:45.119298 28885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:45.120455 28885 hybrid_clock.cc:648] HybridClock initialized: now 1714593705120443 us; error 32 us; skew 500 ppm
May 01 20:01:45 dist-test-slave-tzct krb5kdc[28872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:45.123498 28885 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 20:01:45.124454 28885 webserver.cc:460] Webserver started at http://127.0.0.1:34955/ using document root <none> and password file <none>
I20240501 20:01:45.125047 28885 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:45.125111 28885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:45.125291 28885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:45.127071 28885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "a0b604c5a683436a86075098b9c9efb1"
format_stamp: "Formatted at 2024-05-01 20:01:45 on dist-test-slave-tzct"
server_key: "20f0b5ed86a378ad6a09a092d30d4891"
server_key_iv: "0b4454c04074c5467c7ba4a0a6eb35f9"
server_key_version: "encryptionkey@0"
I20240501 20:01:45.127532 28885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "a0b604c5a683436a86075098b9c9efb1"
format_stamp: "Formatted at 2024-05-01 20:01:45 on dist-test-slave-tzct"
server_key: "20f0b5ed86a378ad6a09a092d30d4891"
server_key_iv: "0b4454c04074c5467c7ba4a0a6eb35f9"
server_key_version: "encryptionkey@0"
I20240501 20:01:45.130832 28885 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 20:01:45.132973 28900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:45.134050 28885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:01:45.134186 28885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "a0b604c5a683436a86075098b9c9efb1"
format_stamp: "Formatted at 2024-05-01 20:01:45 on dist-test-slave-tzct"
server_key: "20f0b5ed86a378ad6a09a092d30d4891"
server_key_iv: "0b4454c04074c5467c7ba4a0a6eb35f9"
server_key_version: "encryptionkey@0"
I20240501 20:01:45.134321 28885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:45.157176 28885 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:45.160535 28885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:45.160764 28885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:45.168076 28885 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37961
I20240501 20:01:45.168290 28952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37961 every 8 connection(s)
I20240501 20:01:45.169164 28885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:45.169466 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 28885
I20240501 20:01:45.169565 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:45.169832 8865 external_mini_cluster.cc:1428] Setting key 0ada9fc7ac89528740238ab8f92762bb
I20240501 20:01:45.171963 28953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
May 01 20:01:45 dist-test-slave-tzct krb5kdc[28872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240501 20:01:45.178212 28953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1: Bootstrap starting.
I20240501 20:01:45.180331 28953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:45.181205 28953 log.cc:824] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:45.182715 28956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:45.171046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40483 (local address 127.0.0.1:37961)
0501 20:01:45.171436 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:45.171446 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:45.171479 (+ 33us) server_negotiation.cc:407] Connection header received
0501 20:01:45.172076 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:45.172097 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:45.172382 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:45.172675 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:45.173799 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:45.174601 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:45.175473 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:45.175665 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:45.178255 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:45.178326 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:45.178342 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:45.178380 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:45.180058 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:45.180551 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:45.180556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:45.180560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:45.180637 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:45.180922 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:45.180924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:45.180925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:45.181269 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:45.181477 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:45.181732 (+ 255us) server_negotiation.cc:299] Negotiation successful
0501 20:01:45.181984 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":119,"threads_started":1}
I20240501 20:01:45.183224 28953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1: No bootstrap required, opened a new log
I20240501 20:01:45.185617 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:01:45.186348 28953 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b604c5a683436a86075098b9c9efb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37961 } }
I20240501 20:01:45.186535 28953 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:45.186570 28953 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b604c5a683436a86075098b9c9efb1, State: Initialized, Role: FOLLOWER
I20240501 20:01:45.187027 28953 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b604c5a683436a86075098b9c9efb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37961 } }
I20240501 20:01:45.187157 28953 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:45.187222 28953 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:45.187333 28953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:45.188287 28953 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b604c5a683436a86075098b9c9efb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37961 } }
I20240501 20:01:45.188616 28953 leader_election.cc:302] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0b604c5a683436a86075098b9c9efb1; no voters:
I20240501 20:01:45.188902 28953 leader_election.cc:288] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:45.188977 28963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:45.189149 28963 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [term 1 LEADER]: Becoming Leader. State: Replica: a0b604c5a683436a86075098b9c9efb1, State: Running, Role: LEADER
I20240501 20:01:45.189518 28963 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b604c5a683436a86075098b9c9efb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37961 } }
I20240501 20:01:45.190093 28953 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:45.191444 28964 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0b604c5a683436a86075098b9c9efb1. Latest consensus state: current_term: 1 leader_uuid: "a0b604c5a683436a86075098b9c9efb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b604c5a683436a86075098b9c9efb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37961 } } }
I20240501 20:01:45.191555 28964 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:45.191895 28965 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0b604c5a683436a86075098b9c9efb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b604c5a683436a86075098b9c9efb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37961 } } }
I20240501 20:01:45.192037 28965 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:45.195238 28956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:45.187458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38763 (local address 127.0.0.1:37961)
0501 20:01:45.187577 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:45.187580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:45.187827 (+ 247us) server_negotiation.cc:407] Connection header received
0501 20:01:45.187968 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:45.187971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:45.188020 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:45.188084 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:45.188908 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:45.189583 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:45.190194 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:45.190402 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:45.191929 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:45.191947 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:45.191951 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:45.191980 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:45.193571 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:45.193929 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:45.193932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:45.193935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:45.193977 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:45.194167 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:45.194170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:45.194172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:45.194335 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:45.194467 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:45.195025 (+ 558us) server_negotiation.cc:299] Negotiation successful
0501 20:01:45.195124 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20240501 20:01:45.195624 28978 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:01:45.195709 28978 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:01:45.195772 28972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:45.196764 28972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:45.202128 28972 catalog_manager.cc:1347] Generated new cluster ID: ef736d1178be4f06926fead63c85b4bc
I20240501 20:01:45.202207 28972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:45.224339 28972 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:45.225649 28972 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:45.237125 28972 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a0b604c5a683436a86075098b9c9efb1: Generated new TSK 0
I20240501 20:01:45.237980 28972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:45.247794 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 28885
2024-05-01T20:01:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1256 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:45 dist-test-slave-tzct krb5kdc[28984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:45 dist-test-slave-tzct krb5kdc[28984](info): set up 2 sockets
May 01 20:01:45 dist-test-slave-tzct krb5kdc[28984](info): commencing operation
krb5kdc: starting...
W20240501 20:01:46.137105 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.870s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:46 dist-test-slave-tzct krb5kdc[28984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 20:01:46.251361 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:46.358698 28997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:46.358975 28997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:46.359054 28997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:46.362175 28997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:46.362250 28997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:46.362280 28997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:46.362304 28997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:46.362334 28997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:46.366547 28997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:46.367480 28997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:46.368332 28997 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:46.375999 29002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:46.376259 29005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:46.376470 29003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:46.377084 28997 server_base.cc:1034] running on GCE node
I20240501 20:01:46.377660 28997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:46.378659 28997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:46.379827 28997 hybrid_clock.cc:648] HybridClock initialized: now 1714593706379801 us; error 43 us; skew 500 ppm
May 01 20:01:46 dist-test-slave-tzct krb5kdc[28984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:46.386577 28997 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 20:01:46.387586 28997 webserver.cc:460] Webserver started at http://127.0.0.1:33545/ using document root <none> and password file <none>
I20240501 20:01:46.388139 28997 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:46.388201 28997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:46.388384 28997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:46.390331 28997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "50825405ec874e6f87a8d83053cb9272"
format_stamp: "Formatted at 2024-05-01 20:01:46 on dist-test-slave-tzct"
server_key: "a1c3674b904f49785eadaee3e6a93ad1"
server_key_iv: "46b7e1b2226aeb4dab38a732fd604d4c"
server_key_version: "encryptionkey@0"
I20240501 20:01:46.390820 28997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "50825405ec874e6f87a8d83053cb9272"
format_stamp: "Formatted at 2024-05-01 20:01:46 on dist-test-slave-tzct"
server_key: "a1c3674b904f49785eadaee3e6a93ad1"
server_key_iv: "46b7e1b2226aeb4dab38a732fd604d4c"
server_key_version: "encryptionkey@0"
I20240501 20:01:46.394263 28997 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 20:01:46.396525 29012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:46.397584 28997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:46.397693 28997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "50825405ec874e6f87a8d83053cb9272"
format_stamp: "Formatted at 2024-05-01 20:01:46 on dist-test-slave-tzct"
server_key: "a1c3674b904f49785eadaee3e6a93ad1"
server_key_iv: "46b7e1b2226aeb4dab38a732fd604d4c"
server_key_version: "encryptionkey@0"
I20240501 20:01:46.397797 28997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:46.434783 28997 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:46.437989 28997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:46.438158 28997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:46.445339 28997 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46419
I20240501 20:01:46.445560 29064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46419 every 8 connection(s)
I20240501 20:01:46.446414 28997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:46.447412 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 28997
I20240501 20:01:46.447525 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:46.447762 8865 external_mini_cluster.cc:1428] Setting key 8be94d61ba656352748784c9cc8310fb
I20240501 20:01:46.449605 29065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:46.456363 29065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272: Bootstrap starting.
I20240501 20:01:46.456636 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:46.449241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46421 (local address 127.0.0.1:46419)
0501 20:01:46.449759 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:46.449772 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:46.449817 (+ 45us) server_negotiation.cc:407] Connection header received
0501 20:01:46.450616 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:46.450644 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:46.451059 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:46.451502 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:46.452022 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:46.453225 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:46.454130 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:46.454417 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:46.454842 (+ 425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:46.454916 (+ 74us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:46.454931 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:46.455427 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:46.455469 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:46.455492 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:46.455608 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:46.455692 (+ 84us) server_negotiation.cc:299] Negotiation successful
0501 20:01:46.455882 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":169,"threads_started":1}
I20240501 20:01:46.458865 29065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:46.459683 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:01:46.459766 29065 log.cc:824] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:46.462038 29065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272: No bootstrap required, opened a new log
I20240501 20:01:46.465478 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:46.461772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51123 (local address 127.0.0.1:46419)
0501 20:01:46.461912 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:46.461915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:46.462092 (+ 177us) server_negotiation.cc:407] Connection header received
0501 20:01:46.462226 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:46.462229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:46.462275 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:46.462343 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:46.462882 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:46.463364 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:46.464232 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:46.464388 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:46.465056 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:46.465074 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:46.465077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:46.465090 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:46.465117 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:46.465127 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:46.465189 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:46.465286 (+ 97us) server_negotiation.cc:299] Negotiation successful
0501 20:01:46.465337 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240501 20:01:46.466158 29065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50825405ec874e6f87a8d83053cb9272" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46419 } }
I20240501 20:01:46.466399 29065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:46.466472 29065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50825405ec874e6f87a8d83053cb9272, State: Initialized, Role: FOLLOWER
I20240501 20:01:46.467011 29065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50825405ec874e6f87a8d83053cb9272" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46419 } }
I20240501 20:01:46.467139 29065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:46.467200 29065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:46.467296 29065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:46.468214 29065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50825405ec874e6f87a8d83053cb9272" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46419 } }
I20240501 20:01:46.468569 29065 leader_election.cc:302] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50825405ec874e6f87a8d83053cb9272; no voters:
I20240501 20:01:46.468922 29065 leader_election.cc:288] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:46.469077 29075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:46.469269 29075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [term 1 LEADER]: Becoming Leader. State: Replica: 50825405ec874e6f87a8d83053cb9272, State: Running, Role: LEADER
I20240501 20:01:46.469558 29075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50825405ec874e6f87a8d83053cb9272" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46419 } }
I20240501 20:01:46.470021 29065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:46.471026 29076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50825405ec874e6f87a8d83053cb9272" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50825405ec874e6f87a8d83053cb9272" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46419 } } }
I20240501 20:01:46.471144 29076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:46.471386 29077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50825405ec874e6f87a8d83053cb9272. Latest consensus state: current_term: 1 leader_uuid: "50825405ec874e6f87a8d83053cb9272" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50825405ec874e6f87a8d83053cb9272" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46419 } } }
I20240501 20:01:46.471475 29077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:46.471535 29084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:46.474524 29084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:46.479903 29084 catalog_manager.cc:1347] Generated new cluster ID: 69074f7e75c841d9a237706f4d2007b2
I20240501 20:01:46.480005 29084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:46.493860 29084 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:46.495245 29084 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:46.512027 29084 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 50825405ec874e6f87a8d83053cb9272: Generated new TSK 0
I20240501 20:01:46.513027 29084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:46.524870 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 28997
2024-05-01T20:01:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:46 dist-test-slave-tzct krb5kdc[29096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:46 dist-test-slave-tzct krb5kdc[29096](info): set up 2 sockets
May 01 20:01:46 dist-test-slave-tzct krb5kdc[29096](info): commencing operation
krb5kdc: starting...
W20240501 20:01:47.413118 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.869s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:47 dist-test-slave-tzct krb5kdc[29096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 20:01:47.528776 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39495
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:47.635169 29109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:47.635435 29109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:47.635509 29109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:47.638926 29109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:47.639011 29109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:47.639042 29109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:47.639066 29109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:47.639094 29109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:47.643378 29109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:47.644304 29109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:47.645193 29109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:47.652091 29114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:47.652365 29115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:47.652894 29117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:47.653353 29109 server_base.cc:1034] running on GCE node
I20240501 20:01:47.653861 29109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:47.654801 29109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:47.655951 29109 hybrid_clock.cc:648] HybridClock initialized: now 1714593707655938 us; error 29 us; skew 500 ppm
May 01 20:01:47 dist-test-slave-tzct krb5kdc[29096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:47.658555 29109 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 20:01:47.659466 29109 webserver.cc:460] Webserver started at http://127.0.0.1:42183/ using document root <none> and password file <none>
I20240501 20:01:47.659976 29109 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:47.660034 29109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:47.660200 29109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:47.662036 29109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "84123ed9c0524b5ab9e365a962ea9d9e"
format_stamp: "Formatted at 2024-05-01 20:01:47 on dist-test-slave-tzct"
server_key: "3e5461ae0dd57d7d3f26560279eafca8"
server_key_iv: "0f6f058fd251c374394ff76412aa2a78"
server_key_version: "encryptionkey@0"
I20240501 20:01:47.662519 29109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "84123ed9c0524b5ab9e365a962ea9d9e"
format_stamp: "Formatted at 2024-05-01 20:01:47 on dist-test-slave-tzct"
server_key: "3e5461ae0dd57d7d3f26560279eafca8"
server_key_iv: "0f6f058fd251c374394ff76412aa2a78"
server_key_version: "encryptionkey@0"
I20240501 20:01:47.665884 29109 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:01:47.668076 29124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:47.669137 29109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:47.669260 29109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "84123ed9c0524b5ab9e365a962ea9d9e"
format_stamp: "Formatted at 2024-05-01 20:01:47 on dist-test-slave-tzct"
server_key: "3e5461ae0dd57d7d3f26560279eafca8"
server_key_iv: "0f6f058fd251c374394ff76412aa2a78"
server_key_version: "encryptionkey@0"
I20240501 20:01:47.669373 29109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:47.674307 29109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:47.677201 29109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:47.677357 29109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:47.683751 29109 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39495
I20240501 20:01:47.684016 29176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39495 every 8 connection(s)
I20240501 20:01:47.684803 29109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:47.687211 29177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:47.692191 29177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e: Bootstrap starting.
I20240501 20:01:47.694103 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 29109
I20240501 20:01:47.694201 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:47.694401 8865 external_mini_cluster.cc:1428] Setting key 147e4b8427ff5757150c7c2853c0d682
I20240501 20:01:47.694583 29177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:47.695271 29177 log.cc:824] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:47.697036 29177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e: No bootstrap required, opened a new log
I20240501 20:01:47.698841 29180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:47.695403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41109 (local address 127.0.0.1:39495)
0501 20:01:47.695819 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:47.695829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:47.695862 (+ 33us) server_negotiation.cc:407] Connection header received
0501 20:01:47.696505 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:47.696522 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:47.696790 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:47.697123 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:47.697864 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:47.697869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:47.697878 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:47.697887 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:47.697962 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:47.697978 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:47.698091 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:47.698175 (+ 84us) server_negotiation.cc:299] Negotiation successful
0501 20:01:47.698304 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":145,"threads_started":1}
I20240501 20:01:47.700115 29177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84123ed9c0524b5ab9e365a962ea9d9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39495 } }
I20240501 20:01:47.700295 29177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:47.700347 29177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84123ed9c0524b5ab9e365a962ea9d9e, State: Initialized, Role: FOLLOWER
I20240501 20:01:47.700814 29177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84123ed9c0524b5ab9e365a962ea9d9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39495 } }
I20240501 20:01:47.700953 29177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:47.701015 29177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:47.701126 29177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:47.701406 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:01:47.702207 29177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84123ed9c0524b5ab9e365a962ea9d9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39495 } }
I20240501 20:01:47.702574 29177 leader_election.cc:302] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84123ed9c0524b5ab9e365a962ea9d9e; no voters:
I20240501 20:01:47.702914 29177 leader_election.cc:288] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:47.703022 29186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:47.703212 29186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [term 1 LEADER]: Becoming Leader. State: Replica: 84123ed9c0524b5ab9e365a962ea9d9e, State: Running, Role: LEADER
I20240501 20:01:47.703951 29186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84123ed9c0524b5ab9e365a962ea9d9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39495 } }
I20240501 20:01:47.704038 29177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:47.705093 29186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84123ed9c0524b5ab9e365a962ea9d9e. Latest consensus state: current_term: 1 leader_uuid: "84123ed9c0524b5ab9e365a962ea9d9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84123ed9c0524b5ab9e365a962ea9d9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39495 } } }
I20240501 20:01:47.705212 29186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:47.705430 29187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84123ed9c0524b5ab9e365a962ea9d9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84123ed9c0524b5ab9e365a962ea9d9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39495 } } }
I20240501 20:01:47.705533 29187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:47.707289 29180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:47.703536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59189 (local address 127.0.0.1:39495)
0501 20:01:47.703658 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:47.703662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:47.705628 (+ 1966us) server_negotiation.cc:407] Connection header received
0501 20:01:47.705854 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:47.705857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:47.705902 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:47.705998 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:47.706832 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:47.706836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:47.706839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:47.706850 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:47.706880 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:47.706892 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:47.706978 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:47.707139 (+ 161us) server_negotiation.cc:299] Negotiation successful
0501 20:01:47.707173 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20240501 20:01:47.708767 29202 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:01:47.708842 29202 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:01:47.709045 29196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:47.709939 29196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:47.715272 29196 catalog_manager.cc:1347] Generated new cluster ID: a860c0724d1946c3a483809d98817c0b
I20240501 20:01:47.715349 29196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:47.731314 29196 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:47.731660 29196 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:47.739414 29196 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 84123ed9c0524b5ab9e365a962ea9d9e: Generated new TSK 0
I20240501 20:01:47.740265 29196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:47.744452 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 29109
2024-05-01T20:01:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:47 dist-test-slave-tzct krb5kdc[29208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:47 dist-test-slave-tzct krb5kdc[29208](info): set up 2 sockets
May 01 20:01:47 dist-test-slave-tzct krb5kdc[29208](info): commencing operation
krb5kdc: starting...
W20240501 20:01:48.644986 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.882s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:48 dist-test-slave-tzct krb5kdc[29208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 20:01:48.755882 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33763
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:48.860850 29221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:48.861088 29221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:48.861147 29221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:48.864152 29221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:48.864216 29221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:48.864239 29221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:48.864271 29221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:48.864300 29221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:48.868304 29221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:48.869155 29221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:48.869956 29221 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:48.876668 29226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:48.876981 29229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:48.877497 29227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:48.878029 29221 server_base.cc:1034] running on GCE node
I20240501 20:01:48.878517 29221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:48.879426 29221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:48.880590 29221 hybrid_clock.cc:648] HybridClock initialized: now 1714593708880572 us; error 39 us; skew 500 ppm
May 01 20:01:48 dist-test-slave-tzct krb5kdc[29208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:48.883314 29221 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 20:01:48.884220 29221 webserver.cc:460] Webserver started at http://127.0.0.1:36443/ using document root <none> and password file <none>
I20240501 20:01:48.884754 29221 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:48.884812 29221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:48.884971 29221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:48.886703 29221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "32860e69ea8b47e8b05be88d30ddee7e"
format_stamp: "Formatted at 2024-05-01 20:01:48 on dist-test-slave-tzct"
server_key: "41e0d05e741cbb717b5c6ac9d8eedfd7"
server_key_iv: "6b15bf7c2b73f518c08d44accfa64939"
server_key_version: "encryptionkey@0"
I20240501 20:01:48.887120 29221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "32860e69ea8b47e8b05be88d30ddee7e"
format_stamp: "Formatted at 2024-05-01 20:01:48 on dist-test-slave-tzct"
server_key: "41e0d05e741cbb717b5c6ac9d8eedfd7"
server_key_iv: "6b15bf7c2b73f518c08d44accfa64939"
server_key_version: "encryptionkey@0"
I20240501 20:01:48.890300 29221 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240501 20:01:48.892421 29236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:48.893409 29221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:48.893530 29221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "32860e69ea8b47e8b05be88d30ddee7e"
format_stamp: "Formatted at 2024-05-01 20:01:48 on dist-test-slave-tzct"
server_key: "41e0d05e741cbb717b5c6ac9d8eedfd7"
server_key_iv: "6b15bf7c2b73f518c08d44accfa64939"
server_key_version: "encryptionkey@0"
I20240501 20:01:48.893613 29221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:48.921885 29221 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:48.924680 29221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:48.924885 29221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:48.931438 29221 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33763
I20240501 20:01:48.931630 29288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33763 every 8 connection(s)
I20240501 20:01:48.932417 29221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:48.934862 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:48.939633 29289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e: Bootstrap starting.
I20240501 20:01:48.941709 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 29221
I20240501 20:01:48.941761 29289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:48.941861 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:48.942092 8865 external_mini_cluster.cc:1428] Setting key 6bcafa745e36915b517640e3f2c4f5fd
I20240501 20:01:48.942446 29289 log.cc:824] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:48.944469 29289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e: No bootstrap required, opened a new log
I20240501 20:01:48.947973 29289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32860e69ea8b47e8b05be88d30ddee7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33763 } }
I20240501 20:01:48.948204 29289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:48.948271 29289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32860e69ea8b47e8b05be88d30ddee7e, State: Initialized, Role: FOLLOWER
I20240501 20:01:48.948712 29289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32860e69ea8b47e8b05be88d30ddee7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33763 } }
I20240501 20:01:48.948872 29289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:48.948926 29289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:48.949026 29289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [term 0 FOLLOWER]: Advancing to term 1
May 01 20:01:48 dist-test-slave-tzct krb5kdc[29208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240501 20:01:48.950034 29289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32860e69ea8b47e8b05be88d30ddee7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33763 } }
I20240501 20:01:48.950356 29289 leader_election.cc:302] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32860e69ea8b47e8b05be88d30ddee7e; no voters:
I20240501 20:01:48.950618 29289 leader_election.cc:288] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:48.950881 29294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:48.951069 29294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [term 1 LEADER]: Becoming Leader. State: Replica: 32860e69ea8b47e8b05be88d30ddee7e, State: Running, Role: LEADER
I20240501 20:01:48.951385 29294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32860e69ea8b47e8b05be88d30ddee7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33763 } }
I20240501 20:01:48.951831 29289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:48.952922 29296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32860e69ea8b47e8b05be88d30ddee7e. Latest consensus state: current_term: 1 leader_uuid: "32860e69ea8b47e8b05be88d30ddee7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32860e69ea8b47e8b05be88d30ddee7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33763 } } }
I20240501 20:01:48.953050 29296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:48.953302 29295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32860e69ea8b47e8b05be88d30ddee7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32860e69ea8b47e8b05be88d30ddee7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33763 } } }
I20240501 20:01:48.953387 29295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:48.953514 29303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:48.955826 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:48.943082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:36159 (local address 127.0.0.1:33763)
0501 20:01:48.943487 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:48.943498 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:48.943535 (+ 37us) server_negotiation.cc:407] Connection header received
0501 20:01:48.944297 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:48.944321 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:48.944668 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:48.945076 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:48.945836 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:48.947044 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:48.947692 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:48.947962 (+ 270us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:48.950266 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:48.950292 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:48.950304 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:48.950348 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:48.952740 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:48.953367 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:48.953373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:48.953379 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:48.953482 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:48.953851 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:48.953854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:48.953856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:48.954305 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:48.954608 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:48.954842 (+ 234us) server_negotiation.cc:299] Negotiation successful
0501 20:01:48.955080 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":120,"threads_started":1}
I20240501 20:01:48.956266 29303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:48.958639 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:01:48.961619 29303 catalog_manager.cc:1347] Generated new cluster ID: ef81803a90774bb4a409d31401a588e0
I20240501 20:01:48.961678 29303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:48.968798 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:48.960681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:52365 (local address 127.0.0.1:33763)
0501 20:01:48.960863 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:48.960867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:48.961096 (+ 229us) server_negotiation.cc:407] Connection header received
0501 20:01:48.961276 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:48.961280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:48.961330 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:48.961380 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:48.962486 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:48.963188 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:48.964020 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:48.964199 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:48.965655 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:48.965671 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:48.965675 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:48.965699 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:48.967224 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:48.967731 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:48.967734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:48.967736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:48.967777 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:48.968028 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:48.968031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:48.968032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:48.968174 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:48.968289 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:48.968527 (+ 238us) server_negotiation.cc:299] Negotiation successful
0501 20:01:48.968610 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20240501 20:01:48.980590 29303 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:48.981700 29303 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:48.990226 29303 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 32860e69ea8b47e8b05be88d30ddee7e: Generated new TSK 0
I20240501 20:01:48.990924 29303 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:48.998477 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 29221
W20240501 20:01:49.003058 29206 connection.cc:535] client connection to 127.0.0.1:33763 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33763): Connection reset by peer (error 104)
2024-05-01T20:01:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:49 dist-test-slave-tzct krb5kdc[29320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:49 dist-test-slave-tzct krb5kdc[29320](info): set up 2 sockets
May 01 20:01:49 dist-test-slave-tzct krb5kdc[29320](info): commencing operation
krb5kdc: starting...
W20240501 20:01:49.889077 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.872s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:49 dist-test-slave-tzct krb5kdc[29320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 20:01:49.997746 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42501
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:50.101370 29333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:50.101624 29333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:50.101701 29333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:50.105036 29333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:50.105116 29333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:50.105147 29333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:50.105171 29333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:50.105197 29333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:50.109495 29333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:50.110395 29333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:50.111229 29333 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:50.118283 29339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:50.118676 29341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:50.119278 29338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:50.119688 29333 server_base.cc:1034] running on GCE node
I20240501 20:01:50.120203 29333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:50.121181 29333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:50.122368 29333 hybrid_clock.cc:648] HybridClock initialized: now 1714593710122351 us; error 31 us; skew 500 ppm
May 01 20:01:50 dist-test-slave-tzct krb5kdc[29320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:50.125404 29333 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 20:01:50.126376 29333 webserver.cc:460] Webserver started at http://127.0.0.1:40869/ using document root <none> and password file <none>
I20240501 20:01:50.126917 29333 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:50.126972 29333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:50.127154 29333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:50.128988 29333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "e2eafc5890fe4fdcaa07d430930effa0"
format_stamp: "Formatted at 2024-05-01 20:01:50 on dist-test-slave-tzct"
server_key: "2b622214ddf819abac867ebe0efae180"
server_key_iv: "4c40a2e340a5fcfef8ee51013e305a0c"
server_key_version: "encryptionkey@0"
I20240501 20:01:50.129451 29333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "e2eafc5890fe4fdcaa07d430930effa0"
format_stamp: "Formatted at 2024-05-01 20:01:50 on dist-test-slave-tzct"
server_key: "2b622214ddf819abac867ebe0efae180"
server_key_iv: "4c40a2e340a5fcfef8ee51013e305a0c"
server_key_version: "encryptionkey@0"
I20240501 20:01:50.132686 29333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:01:50.134919 29348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:50.135874 29333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:50.135993 29333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "e2eafc5890fe4fdcaa07d430930effa0"
format_stamp: "Formatted at 2024-05-01 20:01:50 on dist-test-slave-tzct"
server_key: "2b622214ddf819abac867ebe0efae180"
server_key_iv: "4c40a2e340a5fcfef8ee51013e305a0c"
server_key_version: "encryptionkey@0"
I20240501 20:01:50.136092 29333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:50.150506 29333 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:50.153430 29333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:50.153596 29333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:50.160218 29333 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42501
I20240501 20:01:50.160429 29400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42501 every 8 connection(s)
I20240501 20:01:50.161257 29333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:50.162554 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 29333
I20240501 20:01:50.162657 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:50.162869 8865 external_mini_cluster.cc:1428] Setting key 0148083ef7d2338186ac549424d0cbaa
I20240501 20:01:50.163933 29401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:50.169407 29401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0: Bootstrap starting.
I20240501 20:01:50.170109 29404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:50.164014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:58297 (local address 127.0.0.1:42501)
0501 20:01:50.164400 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:50.164409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:50.164443 (+ 34us) server_negotiation.cc:407] Connection header received
0501 20:01:50.165062 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:50.165080 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:50.165373 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:50.165674 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:50.166217 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:50.166944 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:50.167781 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:50.167979 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:50.168608 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:50.168663 (+ 55us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:50.168673 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:50.169078 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:50.169111 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:50.169135 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:50.169240 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:50.169405 (+ 165us) server_negotiation.cc:299] Negotiation successful
0501 20:01:50.169517 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
I20240501 20:01:50.171397 29401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:50.172130 29401 log.cc:824] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:50.172560 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:01:50.173799 29401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0: No bootstrap required, opened a new log
I20240501 20:01:50.176779 29401 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2eafc5890fe4fdcaa07d430930effa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42501 } }
I20240501 20:01:50.176973 29401 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:50.177032 29401 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2eafc5890fe4fdcaa07d430930effa0, State: Initialized, Role: FOLLOWER
I20240501 20:01:50.177469 29401 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2eafc5890fe4fdcaa07d430930effa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42501 } }
I20240501 20:01:50.177624 29401 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:50.177692 29401 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:50.177806 29401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:50.178692 29401 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2eafc5890fe4fdcaa07d430930effa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42501 } }
I20240501 20:01:50.179068 29401 leader_election.cc:302] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2eafc5890fe4fdcaa07d430930effa0; no voters:
I20240501 20:01:50.179387 29401 leader_election.cc:288] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:50.179462 29411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:50.179674 29411 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [term 1 LEADER]: Becoming Leader. State: Replica: e2eafc5890fe4fdcaa07d430930effa0, State: Running, Role: LEADER
I20240501 20:01:50.179993 29411 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2eafc5890fe4fdcaa07d430930effa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42501 } }
I20240501 20:01:50.180306 29404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:50.174306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:49105 (local address 127.0.0.1:42501)
0501 20:01:50.174398 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:50.174401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:50.176970 (+ 2569us) server_negotiation.cc:407] Connection header received
0501 20:01:50.177137 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:50.177140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:50.177182 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:50.177253 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:50.177729 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:50.178223 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:50.178990 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:50.179120 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:50.179760 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:50.179776 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:50.179779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:50.179791 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:50.179821 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:50.179833 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:50.179890 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:50.180151 (+ 261us) server_negotiation.cc:299] Negotiation successful
0501 20:01:50.180185 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240501 20:01:50.180549 29401 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:50.181958 29413 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2eafc5890fe4fdcaa07d430930effa0. Latest consensus state: current_term: 1 leader_uuid: "e2eafc5890fe4fdcaa07d430930effa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2eafc5890fe4fdcaa07d430930effa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42501 } } }
I20240501 20:01:50.182082 29413 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:50.182359 29412 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2eafc5890fe4fdcaa07d430930effa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2eafc5890fe4fdcaa07d430930effa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42501 } } }
I20240501 20:01:50.182432 29412 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0 [sys.catalog]: This master's current role is: LEADER
W20240501 20:01:50.184929 29426 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:01:50.184998 29426 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:01:50.185122 29423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:50.185989 29423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:50.190740 29423 catalog_manager.cc:1347] Generated new cluster ID: eb1929d22fba4fe596d580f309104a25
I20240501 20:01:50.190809 29423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:50.199498 29423 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:50.200381 29423 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:50.207077 29423 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e2eafc5890fe4fdcaa07d430930effa0: Generated new TSK 0
I20240501 20:01:50.207906 29423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:50.212107 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 29333
W20240501 20:01:50.216480 29318 connection.cc:535] client connection to 127.0.0.1:42501 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42501): Connection reset by peer (error 104)
2024-05-01T20:01:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1213 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:50 dist-test-slave-tzct krb5kdc[29432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:50 dist-test-slave-tzct krb5kdc[29432](info): set up 2 sockets
May 01 20:01:50 dist-test-slave-tzct krb5kdc[29432](info): commencing operation
krb5kdc: starting...
W20240501 20:01:51.101117 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.871s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:51 dist-test-slave-tzct krb5kdc[29432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 20:01:51.210040 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43313
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:51.316489 29445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:51.316773 29445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:51.316839 29445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:51.319906 29445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:51.319970 29445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:51.319993 29445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:51.320010 29445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:51.320026 29445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:51.324141 29445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:51.325021 29445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:51.325834 29445 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:51.332837 29451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:51.333312 29450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:51.333928 29453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:51.334416 29445 server_base.cc:1034] running on GCE node
I20240501 20:01:51.334971 29445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:51.335978 29445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:51.337136 29445 hybrid_clock.cc:648] HybridClock initialized: now 1714593711337112 us; error 38 us; skew 500 ppm
May 01 20:01:51 dist-test-slave-tzct krb5kdc[29432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:51.339845 29445 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 20:01:51.340920 29445 webserver.cc:460] Webserver started at http://127.0.0.1:43739/ using document root <none> and password file <none>
I20240501 20:01:51.341455 29445 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:51.341516 29445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:51.341691 29445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:51.343557 29445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "3092e9fd1e0d440fa3cad4ee8f90bfbe"
format_stamp: "Formatted at 2024-05-01 20:01:51 on dist-test-slave-tzct"
server_key: "7f0ada4aebabbcdb06a9dee850528cd4"
server_key_iv: "1a95b0b7ac8bec41afd61102bc0f906f"
server_key_version: "encryptionkey@0"
I20240501 20:01:51.344050 29445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "3092e9fd1e0d440fa3cad4ee8f90bfbe"
format_stamp: "Formatted at 2024-05-01 20:01:51 on dist-test-slave-tzct"
server_key: "7f0ada4aebabbcdb06a9dee850528cd4"
server_key_iv: "1a95b0b7ac8bec41afd61102bc0f906f"
server_key_version: "encryptionkey@0"
I20240501 20:01:51.347514 29445 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240501 20:01:51.349761 29460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:51.350760 29445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:01:51.350883 29445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "3092e9fd1e0d440fa3cad4ee8f90bfbe"
format_stamp: "Formatted at 2024-05-01 20:01:51 on dist-test-slave-tzct"
server_key: "7f0ada4aebabbcdb06a9dee850528cd4"
server_key_iv: "1a95b0b7ac8bec41afd61102bc0f906f"
server_key_version: "encryptionkey@0"
I20240501 20:01:51.350978 29445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:51.356010 29445 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:51.358959 29445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:51.359113 29445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:51.365722 29445 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43313
I20240501 20:01:51.365957 29512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43313 every 8 connection(s)
I20240501 20:01:51.366688 29445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:51.369203 29513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:51.374163 29513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe: Bootstrap starting.
I20240501 20:01:51.375445 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 29445
I20240501 20:01:51.375555 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:51.375752 8865 external_mini_cluster.cc:1428] Setting key 5520f060c18196f12c83f4c27a78a6fe
I20240501 20:01:51.376482 29513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:51.377466 29513 log.cc:824] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:51.379376 29513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe: No bootstrap required, opened a new log
I20240501 20:01:51.380367 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:51.376765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:55077 (local address 127.0.0.1:43313)
0501 20:01:51.377210 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:51.377223 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:51.377266 (+ 43us) server_negotiation.cc:407] Connection header received
0501 20:01:51.377921 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:51.377941 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:51.378182 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:51.378532 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:51.379310 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:51.379314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:51.379324 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:51.379334 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:51.379373 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:51.379387 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:51.379524 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:51.379669 (+ 145us) server_negotiation.cc:299] Negotiation successful
0501 20:01:51.379800 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":130,"threads_started":1}
I20240501 20:01:51.382534 29513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3092e9fd1e0d440fa3cad4ee8f90bfbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } }
I20240501 20:01:51.382725 29513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:51.382828 29513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3092e9fd1e0d440fa3cad4ee8f90bfbe, State: Initialized, Role: FOLLOWER
I20240501 20:01:51.382899 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:01:51.383261 29513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3092e9fd1e0d440fa3cad4ee8f90bfbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } }
I20240501 20:01:51.383384 29513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:51.383441 29513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:51.383540 29513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:51.384429 29513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3092e9fd1e0d440fa3cad4ee8f90bfbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } }
I20240501 20:01:51.384822 29513 leader_election.cc:302] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3092e9fd1e0d440fa3cad4ee8f90bfbe; no voters:
I20240501 20:01:51.385164 29513 leader_election.cc:288] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:51.385255 29523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:51.385506 29523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [term 1 LEADER]: Becoming Leader. State: Replica: 3092e9fd1e0d440fa3cad4ee8f90bfbe, State: Running, Role: LEADER
I20240501 20:01:51.386220 29523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3092e9fd1e0d440fa3cad4ee8f90bfbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } }
I20240501 20:01:51.386410 29513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:51.387713 29524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3092e9fd1e0d440fa3cad4ee8f90bfbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3092e9fd1e0d440fa3cad4ee8f90bfbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } } }
I20240501 20:01:51.387835 29524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:51.388092 29524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3092e9fd1e0d440fa3cad4ee8f90bfbe. Latest consensus state: current_term: 1 leader_uuid: "3092e9fd1e0d440fa3cad4ee8f90bfbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3092e9fd1e0d440fa3cad4ee8f90bfbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } } }
I20240501 20:01:51.388255 29524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:51.388238 29532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:51.390873 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:51.384801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:39237 (local address 127.0.0.1:43313)
0501 20:01:51.384913 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:51.384917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:51.385089 (+ 172us) server_negotiation.cc:407] Connection header received
0501 20:01:51.385212 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:51.385215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:51.385253 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:51.385321 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:51.390427 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:51.390432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:51.390435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:51.390450 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:51.390484 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:51.390498 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:51.390599 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:51.390702 (+ 103us) server_negotiation.cc:299] Negotiation successful
0501 20:01:51.390744 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240501 20:01:51.391323 29532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:51.396219 29532 catalog_manager.cc:1347] Generated new cluster ID: ecb73445187a4b938c79b1f1fbb668ba
I20240501 20:01:51.396284 29532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:51.411922 29532 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:51.412261 29532 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:51.430599 29532 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3092e9fd1e0d440fa3cad4ee8f90bfbe: Generated new TSK 0
I20240501 20:01:51.431444 29532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:51.439538 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 29445
2024-05-01T20:01:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1227 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:51 dist-test-slave-tzct krb5kdc[29544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:51 dist-test-slave-tzct krb5kdc[29544](info): set up 2 sockets
May 01 20:01:51 dist-test-slave-tzct krb5kdc[29544](info): commencing operation
krb5kdc: starting...
W20240501 20:01:52.344990 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.886s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:52 dist-test-slave-tzct krb5kdc[29544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 20:01:52.457800 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39929
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:52.560822 29557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:52.561065 29557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:52.561139 29557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:52.564138 29557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:52.564203 29557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:52.564234 29557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:52.564256 29557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:52.564283 29557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:52.568295 29557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39929
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:52.569177 29557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:52.569990 29557 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:52.576761 29565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:52.577018 29562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:52.577484 29563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:52.577952 29557 server_base.cc:1034] running on GCE node
I20240501 20:01:52.578393 29557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:52.579241 29557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:52.580425 29557 hybrid_clock.cc:648] HybridClock initialized: now 1714593712580407 us; error 31 us; skew 500 ppm
May 01 20:01:52 dist-test-slave-tzct krb5kdc[29544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:52.582856 29557 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 20:01:52.583736 29557 webserver.cc:460] Webserver started at http://127.0.0.1:45349/ using document root <none> and password file <none>
I20240501 20:01:52.584228 29557 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:52.584280 29557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:52.584476 29557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:52.586226 29557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "bb46bf7aa8774a00bbeb4a9989ac5fd0"
format_stamp: "Formatted at 2024-05-01 20:01:52 on dist-test-slave-tzct"
server_key: "2d3aa3521f24c9edf33e5de9dec5fe76"
server_key_iv: "6f975547443ac1e2ecc231ed4e6e11b5"
server_key_version: "encryptionkey@0"
I20240501 20:01:52.586675 29557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "bb46bf7aa8774a00bbeb4a9989ac5fd0"
format_stamp: "Formatted at 2024-05-01 20:01:52 on dist-test-slave-tzct"
server_key: "2d3aa3521f24c9edf33e5de9dec5fe76"
server_key_iv: "6f975547443ac1e2ecc231ed4e6e11b5"
server_key_version: "encryptionkey@0"
I20240501 20:01:52.589788 29557 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:01:52.591764 29572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:52.592917 29557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:52.593040 29557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "bb46bf7aa8774a00bbeb4a9989ac5fd0"
format_stamp: "Formatted at 2024-05-01 20:01:52 on dist-test-slave-tzct"
server_key: "2d3aa3521f24c9edf33e5de9dec5fe76"
server_key_iv: "6f975547443ac1e2ecc231ed4e6e11b5"
server_key_version: "encryptionkey@0"
I20240501 20:01:52.593125 29557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:52.625500 29557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:52.628217 29557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:52.628392 29557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:52.635337 29557 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39929
I20240501 20:01:52.635550 29624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39929 every 8 connection(s)
I20240501 20:01:52.636284 29557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:52.638684 29625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:52.642858 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 29557
I20240501 20:01:52.642966 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:52.643177 8865 external_mini_cluster.cc:1428] Setting key 07108978350ee3c7d91477c3f4efd45c
I20240501 20:01:52.643363 29625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0: Bootstrap starting.
I20240501 20:01:52.645534 29625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:52.646320 29625 log.cc:824] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:52.648098 29625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0: No bootstrap required, opened a new log
May 01 20:01:52 dist-test-slave-tzct krb5kdc[29544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240501 20:01:52.651551 29625 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb46bf7aa8774a00bbeb4a9989ac5fd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } }
I20240501 20:01:52.651732 29625 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:52.651793 29625 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb46bf7aa8774a00bbeb4a9989ac5fd0, State: Initialized, Role: FOLLOWER
I20240501 20:01:52.652233 29625 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb46bf7aa8774a00bbeb4a9989ac5fd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } }
I20240501 20:01:52.652354 29625 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:52.652408 29625 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:52.652529 29625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:52.653380 29625 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb46bf7aa8774a00bbeb4a9989ac5fd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } }
I20240501 20:01:52.653728 29625 leader_election.cc:302] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb46bf7aa8774a00bbeb4a9989ac5fd0; no voters:
I20240501 20:01:52.654013 29625 leader_election.cc:288] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:52.654157 29630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:52.654354 29630 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [term 1 LEADER]: Becoming Leader. State: Replica: bb46bf7aa8774a00bbeb4a9989ac5fd0, State: Running, Role: LEADER
I20240501 20:01:52.654686 29630 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb46bf7aa8774a00bbeb4a9989ac5fd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } }
I20240501 20:01:52.655042 29625 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:52.655843 29630 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb46bf7aa8774a00bbeb4a9989ac5fd0. Latest consensus state: current_term: 1 leader_uuid: "bb46bf7aa8774a00bbeb4a9989ac5fd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb46bf7aa8774a00bbeb4a9989ac5fd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } } }
I20240501 20:01:52.655997 29630 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:52.656193 29631 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb46bf7aa8774a00bbeb4a9989ac5fd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb46bf7aa8774a00bbeb4a9989ac5fd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39929 } } }
I20240501 20:01:52.656292 29631 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:52.656324 29628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:52.644189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:59181 (local address 127.0.0.1:39929)
0501 20:01:52.644544 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:52.644552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:52.644585 (+ 33us) server_negotiation.cc:407] Connection header received
0501 20:01:52.645182 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:52.645199 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:52.645480 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:52.645778 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:52.646740 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:52.647607 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:52.648448 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:52.648636 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:52.651104 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:52.651120 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:52.651132 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:52.651164 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:52.652981 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:52.653476 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:52.653481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:52.653487 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:52.653565 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:52.653849 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:52.653853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:52.653855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:52.654275 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:52.655174 (+ 899us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:52.655345 (+ 171us) server_negotiation.cc:299] Negotiation successful
0501 20:01:52.655545 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":111,"threads_started":1}
I20240501 20:01:52.656376 29639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:52.659358 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:01:52.660563 29639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:52.666879 29639 catalog_manager.cc:1347] Generated new cluster ID: 5d41415b962d4f9bb699ac4faf624d59
I20240501 20:01:52.666958 29639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:52.670356 29628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:52.661541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:46607 (local address 127.0.0.1:39929)
0501 20:01:52.661655 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:52.661659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:52.662599 (+ 940us) server_negotiation.cc:407] Connection header received
0501 20:01:52.662730 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:52.662733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:52.662787 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:52.662877 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:52.663903 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:52.664642 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:52.665307 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:52.665516 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:52.666954 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:52.666972 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:52.666976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:52.667004 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:52.668808 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:52.669283 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:52.669286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:52.669288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:52.669343 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:52.669582 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:52.669585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:52.669587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:52.669758 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:52.669900 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:52.670100 (+ 200us) server_negotiation.cc:299] Negotiation successful
0501 20:01:52.670194 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240501 20:01:52.675236 29639 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:52.676116 29639 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:52.684424 29639 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bb46bf7aa8774a00bbeb4a9989ac5fd0: Generated new TSK 0
I20240501 20:01:52.685161 29639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:52.688699 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 29557
W20240501 20:01:52.693068 29542 connection.cc:535] client connection to 127.0.0.1:39929 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39929): Connection reset by peer (error 104)
2024-05-01T20:01:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1248 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:52 dist-test-slave-tzct krb5kdc[29656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:52 dist-test-slave-tzct krb5kdc[29656](info): set up 2 sockets
May 01 20:01:52 dist-test-slave-tzct krb5kdc[29656](info): commencing operation
krb5kdc: starting...
W20240501 20:01:53.589051 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.882s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:53 dist-test-slave-tzct krb5kdc[29656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 20:01:53.699465 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42145
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:53.802429 29669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:53.802677 29669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:53.802750 29669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:53.805789 29669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:53.805850 29669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:53.805881 29669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:53.805913 29669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:53.805949 29669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:53.809931 29669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42145
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:53.810779 29669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:53.811586 29669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:53.818392 29677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:53.818789 29674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:53.819095 29675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:53.819653 29669 server_base.cc:1034] running on GCE node
I20240501 20:01:53.820127 29669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:53.821039 29669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:53.822186 29669 hybrid_clock.cc:648] HybridClock initialized: now 1714593713822177 us; error 24 us; skew 500 ppm
May 01 20:01:53 dist-test-slave-tzct krb5kdc[29656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:53.824738 29669 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 20:01:53.825590 29669 webserver.cc:460] Webserver started at http://127.0.0.1:43013/ using document root <none> and password file <none>
I20240501 20:01:53.826074 29669 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:53.826126 29669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:53.826293 29669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:53.827991 29669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "9c2c870fd7c442ed9a50b2316072fc9d"
format_stamp: "Formatted at 2024-05-01 20:01:53 on dist-test-slave-tzct"
server_key: "d1136c7d7731417f62d78db01c2e443b"
server_key_iv: "ef53c4ebcf00a7020eb3f7f3f2342654"
server_key_version: "encryptionkey@0"
I20240501 20:01:53.828455 29669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "9c2c870fd7c442ed9a50b2316072fc9d"
format_stamp: "Formatted at 2024-05-01 20:01:53 on dist-test-slave-tzct"
server_key: "d1136c7d7731417f62d78db01c2e443b"
server_key_iv: "ef53c4ebcf00a7020eb3f7f3f2342654"
server_key_version: "encryptionkey@0"
I20240501 20:01:53.831614 29669 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 20:01:53.833681 29684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:53.834679 29669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:01:53.834793 29669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "9c2c870fd7c442ed9a50b2316072fc9d"
format_stamp: "Formatted at 2024-05-01 20:01:53 on dist-test-slave-tzct"
server_key: "d1136c7d7731417f62d78db01c2e443b"
server_key_iv: "ef53c4ebcf00a7020eb3f7f3f2342654"
server_key_version: "encryptionkey@0"
I20240501 20:01:53.834879 29669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:53.858371 29669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:53.861367 29669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:53.861533 29669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:53.867837 29669 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42145
I20240501 20:01:53.868055 29736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42145 every 8 connection(s)
I20240501 20:01:53.868819 29669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:53.871186 29737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:53.874195 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 29669
I20240501 20:01:53.874321 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:53.874581 8865 external_mini_cluster.cc:1428] Setting key fb3946575d1b6b5548fda79a36046e11
I20240501 20:01:53.876405 29737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d: Bootstrap starting.
I20240501 20:01:53.878338 29737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:53.878983 29737 log.cc:824] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:53.880820 29737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d: No bootstrap required, opened a new log
I20240501 20:01:53.881769 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:53.875772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:41521 (local address 127.0.0.1:42145)
0501 20:01:53.876172 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:53.876181 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:53.876214 (+ 33us) server_negotiation.cc:407] Connection header received
0501 20:01:53.876809 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:53.876829 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:53.877120 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:53.877428 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:53.877874 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:53.878646 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:53.879308 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:53.879568 (+ 260us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:53.880109 (+ 541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:53.880128 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:53.880141 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:53.880577 (+ 436us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:53.880617 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:53.880639 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:53.880824 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:53.880984 (+ 160us) server_negotiation.cc:299] Negotiation successful
0501 20:01:53.881093 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":132,"threads_started":1}
I20240501 20:01:53.883936 29737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2c870fd7c442ed9a50b2316072fc9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42145 } }
I20240501 20:01:53.884109 29737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:53.884166 29737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2c870fd7c442ed9a50b2316072fc9d, State: Initialized, Role: FOLLOWER
I20240501 20:01:53.884299 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:01:53.884596 29737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2c870fd7c442ed9a50b2316072fc9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42145 } }
I20240501 20:01:53.884775 29737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:53.884842 29737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:53.884938 29737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:53.885973 29737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2c870fd7c442ed9a50b2316072fc9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42145 } }
I20240501 20:01:53.886317 29737 leader_election.cc:302] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c2c870fd7c442ed9a50b2316072fc9d; no voters:
I20240501 20:01:53.886646 29737 leader_election.cc:288] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:53.886736 29747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:53.886991 29747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [term 1 LEADER]: Becoming Leader. State: Replica: 9c2c870fd7c442ed9a50b2316072fc9d, State: Running, Role: LEADER
I20240501 20:01:53.887331 29747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2c870fd7c442ed9a50b2316072fc9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42145 } }
I20240501 20:01:53.887738 29737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:53.889356 29749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c2c870fd7c442ed9a50b2316072fc9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2c870fd7c442ed9a50b2316072fc9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42145 } } }
I20240501 20:01:53.889478 29749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:53.889994 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:53.886325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:46237 (local address 127.0.0.1:42145)
0501 20:01:53.886468 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:53.886471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:53.886581 (+ 110us) server_negotiation.cc:407] Connection header received
0501 20:01:53.886679 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:53.886682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:53.886728 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:53.886810 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:53.887274 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:53.887891 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:53.888647 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:53.888845 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:53.889514 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:53.889531 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:53.889534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:53.889546 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:53.889576 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:53.889588 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:53.889664 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:53.889826 (+ 162us) server_negotiation.cc:299] Negotiation successful
0501 20:01:53.889858 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240501 20:01:53.890219 29761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:53.889966 29748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c2c870fd7c442ed9a50b2316072fc9d. Latest consensus state: current_term: 1 leader_uuid: "9c2c870fd7c442ed9a50b2316072fc9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2c870fd7c442ed9a50b2316072fc9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42145 } } }
I20240501 20:01:53.890528 29748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:53.892560 29761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:53.897485 29761 catalog_manager.cc:1347] Generated new cluster ID: 45bd5d5260604a09b174061f210b2852
I20240501 20:01:53.897550 29761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:53.921715 29761 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:53.922950 29761 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:53.930745 29761 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9c2c870fd7c442ed9a50b2316072fc9d: Generated new TSK 0
I20240501 20:01:53.931533 29761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:53.937218 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 29669
W20240501 20:01:53.941695 29654 connection.cc:535] client connection to 127.0.0.1:42145 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42145): Connection reset by peer (error 104)
2024-05-01T20:01:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1248 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:53 dist-test-slave-tzct krb5kdc[29768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:53 dist-test-slave-tzct krb5kdc[29768](info): set up 2 sockets
May 01 20:01:53 dist-test-slave-tzct krb5kdc[29768](info): commencing operation
krb5kdc: starting...
W20240501 20:01:54.824957 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.870s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:54 dist-test-slave-tzct krb5kdc[29768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240501 20:01:54.935781 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40397
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:55.042330 29781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:55.042591 29781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:55.042650 29781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:55.045845 29781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:55.045920 29781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:55.045943 29781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:55.045960 29781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:55.045976 29781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:55.050370 29781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40397
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:55.051316 29781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:55.052186 29781 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:55.059615 29789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:55.059652 29786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:55.060122 29787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:55.060662 29781 server_base.cc:1034] running on GCE node
I20240501 20:01:55.061249 29781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:55.062245 29781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:55.063405 29781 hybrid_clock.cc:648] HybridClock initialized: now 1714593715063393 us; error 31 us; skew 500 ppm
May 01 20:01:55 dist-test-slave-tzct krb5kdc[29768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:55.066071 29781 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240501 20:01:55.067047 29781 webserver.cc:460] Webserver started at http://127.0.0.1:39853/ using document root <none> and password file <none>
I20240501 20:01:55.067577 29781 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:55.067633 29781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:55.067804 29781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:55.069648 29781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "5fdffbb784f2485cb8f9393ff30096b8"
format_stamp: "Formatted at 2024-05-01 20:01:55 on dist-test-slave-tzct"
server_key: "433061c516a43bb8a1e77bebb2275ded"
server_key_iv: "7be05bbec595654b7829d1a6a9a8f51c"
server_key_version: "encryptionkey@0"
I20240501 20:01:55.070133 29781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "5fdffbb784f2485cb8f9393ff30096b8"
format_stamp: "Formatted at 2024-05-01 20:01:55 on dist-test-slave-tzct"
server_key: "433061c516a43bb8a1e77bebb2275ded"
server_key_iv: "7be05bbec595654b7829d1a6a9a8f51c"
server_key_version: "encryptionkey@0"
I20240501 20:01:55.073544 29781 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:01:55.075747 29796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:55.076882 29781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:55.077014 29781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "5fdffbb784f2485cb8f9393ff30096b8"
format_stamp: "Formatted at 2024-05-01 20:01:55 on dist-test-slave-tzct"
server_key: "433061c516a43bb8a1e77bebb2275ded"
server_key_iv: "7be05bbec595654b7829d1a6a9a8f51c"
server_key_version: "encryptionkey@0"
I20240501 20:01:55.077117 29781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:55.082216 29781 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:55.085361 29781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:55.085525 29781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:55.092101 29781 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40397
I20240501 20:01:55.092308 29848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40397 every 8 connection(s)
I20240501 20:01:55.093129 29781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:55.095594 29849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:55.100596 29849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8: Bootstrap starting.
I20240501 20:01:55.101281 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 29781
I20240501 20:01:55.101404 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:55.101692 8865 external_mini_cluster.cc:1428] Setting key 691a4bef3c8e11928bcd51c1980d77c7
I20240501 20:01:55.103349 29849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:55.104199 29849 log.cc:824] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:55.106458 29849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8: No bootstrap required, opened a new log
I20240501 20:01:55.106622 29852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:55.102754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:53559 (local address 127.0.0.1:40397)
0501 20:01:55.103200 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:55.103213 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:55.103252 (+ 39us) server_negotiation.cc:407] Connection header received
0501 20:01:55.104038 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:55.104064 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:55.104386 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:55.104792 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:55.105613 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:55.105617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:55.105628 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:55.105638 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:55.105675 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:55.105690 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:55.105789 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:55.105891 (+ 102us) server_negotiation.cc:299] Negotiation successful
0501 20:01:55.106018 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":125,"threads_started":1}
I20240501 20:01:55.109294 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:01:55.109814 29849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdffbb784f2485cb8f9393ff30096b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40397 } }
I20240501 20:01:55.110006 29849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:55.110062 29849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fdffbb784f2485cb8f9393ff30096b8, State: Initialized, Role: FOLLOWER
I20240501 20:01:55.110445 29849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdffbb784f2485cb8f9393ff30096b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40397 } }
I20240501 20:01:55.110558 29849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:55.110620 29849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:55.110714 29849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:55.111567 29849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdffbb784f2485cb8f9393ff30096b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40397 } }
I20240501 20:01:55.111918 29849 leader_election.cc:302] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fdffbb784f2485cb8f9393ff30096b8; no voters:
I20240501 20:01:55.112187 29849 leader_election.cc:288] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:55.112291 29859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:55.112748 29859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [term 1 LEADER]: Becoming Leader. State: Replica: 5fdffbb784f2485cb8f9393ff30096b8, State: Running, Role: LEADER
I20240501 20:01:55.113050 29859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdffbb784f2485cb8f9393ff30096b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40397 } }
I20240501 20:01:55.113234 29852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:55.111247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:58981 (local address 127.0.0.1:40397)
0501 20:01:55.111350 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:55.111354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:55.111586 (+ 232us) server_negotiation.cc:407] Connection header received
0501 20:01:55.111727 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:55.111730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:55.111769 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:55.111850 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:55.112659 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:55.112663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:55.112665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:55.112676 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:55.112708 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:55.112745 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:55.112825 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:55.113085 (+ 260us) server_negotiation.cc:299] Negotiation successful
0501 20:01:55.113123 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240501 20:01:55.113319 29849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:55.114223 29859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fdffbb784f2485cb8f9393ff30096b8. Latest consensus state: current_term: 1 leader_uuid: "5fdffbb784f2485cb8f9393ff30096b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdffbb784f2485cb8f9393ff30096b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40397 } } }
I20240501 20:01:55.114382 29859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:55.114588 29860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fdffbb784f2485cb8f9393ff30096b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdffbb784f2485cb8f9393ff30096b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40397 } } }
I20240501 20:01:55.114717 29860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:55.115079 29868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:55.117746 29868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:55.122809 29868 catalog_manager.cc:1347] Generated new cluster ID: ef7344e8f655452caa9a81e49b411ca0
I20240501 20:01:55.122902 29868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:55.141911 29868 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:55.142227 29868 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:55.151950 29868 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5fdffbb784f2485cb8f9393ff30096b8: Generated new TSK 0
I20240501 20:01:55.152652 29868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:55.157155 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 29781
2024-05-01T20:01:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:55 dist-test-slave-tzct krb5kdc[29880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:55 dist-test-slave-tzct krb5kdc[29880](info): set up 2 sockets
May 01 20:01:55 dist-test-slave-tzct krb5kdc[29880](info): commencing operation
krb5kdc: starting...
W20240501 20:01:56.053218 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.878s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:56 dist-test-slave-tzct krb5kdc[29880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 20:01:56.169557 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:39659
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:41619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:39659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:56.276165 29893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:56.276424 29893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:56.276501 29893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:56.279582 29893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:56.279652 29893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:56.279682 29893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:56.279707 29893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:56.279739 29893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:56.283862 29893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:41619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:39659
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:39659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:56.284787 29893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:56.285607 29893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:56.293283 29898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:56.293341 29901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:56.293655 29899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:56.294220 29893 server_base.cc:1034] running on GCE node
I20240501 20:01:56.294723 29893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:56.295660 29893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:56.296788 29893 hybrid_clock.cc:648] HybridClock initialized: now 1714593716296765 us; error 37 us; skew 500 ppm
May 01 20:01:56 dist-test-slave-tzct krb5kdc[29880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593716, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:56.299374 29893 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:01:56.300326 29893 webserver.cc:460] Webserver started at http://127.8.168.126:40391/ using document root <none> and password file <none>
I20240501 20:01:56.300901 29893 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:56.300967 29893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:56.301153 29893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:56.302990 29893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "c6560f2c3bb84510a73002be0549286a"
format_stamp: "Formatted at 2024-05-01 20:01:56 on dist-test-slave-tzct"
server_key: "acd8dbfee68048e55d465e6b1c2908f0"
server_key_iv: "8ccfa14fdde8f0310beebf7203ccccab"
server_key_version: "encryptionkey@0"
I20240501 20:01:56.303471 29893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "c6560f2c3bb84510a73002be0549286a"
format_stamp: "Formatted at 2024-05-01 20:01:56 on dist-test-slave-tzct"
server_key: "acd8dbfee68048e55d465e6b1c2908f0"
server_key_iv: "8ccfa14fdde8f0310beebf7203ccccab"
server_key_version: "encryptionkey@0"
I20240501 20:01:56.306910 29893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 20:01:56.309167 29908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:56.310230 29893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:56.310365 29893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "c6560f2c3bb84510a73002be0549286a"
format_stamp: "Formatted at 2024-05-01 20:01:56 on dist-test-slave-tzct"
server_key: "acd8dbfee68048e55d465e6b1c2908f0"
server_key_iv: "8ccfa14fdde8f0310beebf7203ccccab"
server_key_version: "encryptionkey@0"
I20240501 20:01:56.310463 29893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:56.326524 29893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:56.329619 29893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:56.329794 29893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:56.336544 29893 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:39659
I20240501 20:01:56.336804 29960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:39659 every 8 connection(s)
I20240501 20:01:56.337638 29893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:56.340077 29961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:56.345043 29961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a: Bootstrap starting.
I20240501 20:01:56.345088 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 29893
I20240501 20:01:56.345185 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:56.345413 8865 external_mini_cluster.cc:1428] Setting key 86f2f1d4ccaa62cf776c7441360322da
I20240501 20:01:56.347430 29961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:56.348259 29961 log.cc:824] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:56.350211 29961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a: No bootstrap required, opened a new log
May 01 20:01:56 dist-test-slave-tzct krb5kdc[29880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:01:56.353487 29961 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6560f2c3bb84510a73002be0549286a" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39659 } }
I20240501 20:01:56.353669 29961 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:56.353814 29961 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6560f2c3bb84510a73002be0549286a, State: Initialized, Role: FOLLOWER
I20240501 20:01:56.354280 29961 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6560f2c3bb84510a73002be0549286a" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39659 } }
I20240501 20:01:56.354436 29961 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:56.354503 29961 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:56.354643 29961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:56.355412 29961 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6560f2c3bb84510a73002be0549286a" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39659 } }
I20240501 20:01:56.355710 29961 leader_election.cc:302] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6560f2c3bb84510a73002be0549286a; no voters:
I20240501 20:01:56.355958 29961 leader_election.cc:288] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:56.356070 29966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:56.356264 29966 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [term 1 LEADER]: Becoming Leader. State: Replica: c6560f2c3bb84510a73002be0549286a, State: Running, Role: LEADER
I20240501 20:01:56.356535 29966 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6560f2c3bb84510a73002be0549286a" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39659 } }
I20240501 20:01:56.357081 29961 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:56.358013 29967 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6560f2c3bb84510a73002be0549286a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6560f2c3bb84510a73002be0549286a" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39659 } } }
I20240501 20:01:56.358145 29967 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:56.358605 29968 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6560f2c3bb84510a73002be0549286a. Latest consensus state: current_term: 1 leader_uuid: "c6560f2c3bb84510a73002be0549286a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6560f2c3bb84510a73002be0549286a" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 39659 } } }
I20240501 20:01:56.358736 29968 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:56.359238 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:56.346529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42783 (local address 127.8.168.126:39659)
0501 20:01:56.346966 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:56.346976 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:56.347017 (+ 41us) server_negotiation.cc:407] Connection header received
0501 20:01:56.347807 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:56.347832 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:56.348205 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:56.348603 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:56.349489 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:56.350629 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:56.351263 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:56.351537 (+ 274us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:56.353740 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:56.353761 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:56.353773 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:56.353814 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:56.355602 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:56.356199 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:56.356205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:56.356210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:56.356305 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:56.356643 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:56.356646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:56.356648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:56.357110 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:56.357339 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:56.358219 (+ 880us) server_negotiation.cc:299] Negotiation successful
0501 20:01:56.358443 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":115,"threads_started":1}
W20240501 20:01:56.361680 29981 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:01:56.361744 29981 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:01:56.361800 29975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:56.362435 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:01:56.362557 29975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:56.368166 29975 catalog_manager.cc:1347] Generated new cluster ID: 6316a7a55ce74075bbf32847680b394c
I20240501 20:01:56.368232 29975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:56.372129 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:56.364506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48011 (local address 127.8.168.126:39659)
0501 20:01:56.364620 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:56.364623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:56.364935 (+ 312us) server_negotiation.cc:407] Connection header received
0501 20:01:56.365048 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:56.365051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:56.365100 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:56.365183 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:56.366022 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:56.366787 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:56.367368 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:56.367554 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:56.368852 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:56.368869 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:56.368872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:01:56.368908 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:56.370546 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:56.371027 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:56.371030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:56.371032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:56.371085 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:01:56.371333 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:01:56.371336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:01:56.371338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:01:56.371518 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:56.371663 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:56.371861 (+ 198us) server_negotiation.cc:299] Negotiation successful
0501 20:01:56.371967 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240501 20:01:56.389760 29975 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:56.390903 29975 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:56.397665 29975 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c6560f2c3bb84510a73002be0549286a: Generated new TSK 0
I20240501 20:01:56.398514 29975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:56.405989 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 29893
W20240501 20:01:56.410535 29878 connection.cc:535] client connection to 127.8.168.126:39659 recv error: Network error: failed to read from TLS socket (remote: 127.8.168.126:39659): Connection reset by peer (error 104)
2024-05-01T20:01:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1248 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:56 dist-test-slave-tzct krb5kdc[29992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:56 dist-test-slave-tzct krb5kdc[29992](info): set up 2 sockets
May 01 20:01:56 dist-test-slave-tzct krb5kdc[29992](info): commencing operation
krb5kdc: starting...
W20240501 20:01:57.305258 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.881s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:57 dist-test-slave-tzct krb5kdc[29992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 20:01:57.420114 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33393
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:33393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:57.524917 30005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:57.525151 30005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:57.525224 30005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:57.528188 30005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:57.528245 30005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:57.528275 30005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:57.528299 30005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:57.528338 30005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:57.532263 30005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:33393
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:57.533129 30005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:57.533941 30005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:57.540777 30011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:57.541139 30013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:57.541594 30010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:57.542025 30005 server_base.cc:1034] running on GCE node
I20240501 20:01:57.542469 30005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:57.543355 30005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:57.544507 30005 hybrid_clock.cc:648] HybridClock initialized: now 1714593717544492 us; error 27 us; skew 500 ppm
May 01 20:01:57 dist-test-slave-tzct krb5kdc[29992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593717, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:57.547091 30005 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:01:57.548017 30005 webserver.cc:460] Webserver started at http://127.8.168.126:43915/ using document root <none> and password file <none>
I20240501 20:01:57.548528 30005 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:57.548585 30005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:57.548801 30005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:57.550592 30005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "e1cf180d5c0343e7813878fb1931ef41"
format_stamp: "Formatted at 2024-05-01 20:01:57 on dist-test-slave-tzct"
server_key: "e0015bedda992200af9ebb29ade68ad6"
server_key_iv: "5812442d45ae4f9cc39b019a474b2147"
server_key_version: "encryptionkey@0"
I20240501 20:01:57.551072 30005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "e1cf180d5c0343e7813878fb1931ef41"
format_stamp: "Formatted at 2024-05-01 20:01:57 on dist-test-slave-tzct"
server_key: "e0015bedda992200af9ebb29ade68ad6"
server_key_iv: "5812442d45ae4f9cc39b019a474b2147"
server_key_version: "encryptionkey@0"
I20240501 20:01:57.554369 30005 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 20:01:57.556821 30020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:57.557842 30005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:57.557977 30005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "e1cf180d5c0343e7813878fb1931ef41"
format_stamp: "Formatted at 2024-05-01 20:01:57 on dist-test-slave-tzct"
server_key: "e0015bedda992200af9ebb29ade68ad6"
server_key_iv: "5812442d45ae4f9cc39b019a474b2147"
server_key_version: "encryptionkey@0"
I20240501 20:01:57.558066 30005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:57.575176 30005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:57.578048 30005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:57.578228 30005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:57.585171 30005 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:33393
I20240501 20:01:57.585417 30072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:33393 every 8 connection(s)
I20240501 20:01:57.586215 30005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:57.588557 30073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:57.593353 30073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41: Bootstrap starting.
I20240501 20:01:57.595458 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 30005
I20240501 20:01:57.595510 30073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:57.595588 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:57.595831 8865 external_mini_cluster.cc:1428] Setting key ca2b71c7f0b3082a85b4910387cca0fc
I20240501 20:01:57.596277 30073 log.cc:824] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:57.598091 30073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41: No bootstrap required, opened a new log
I20240501 20:01:57.600982 30073 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cf180d5c0343e7813878fb1931ef41" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33393 } }
I20240501 20:01:57.601171 30073 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:57.601220 30073 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cf180d5c0343e7813878fb1931ef41, State: Initialized, Role: FOLLOWER
I20240501 20:01:57.601629 30073 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cf180d5c0343e7813878fb1931ef41" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33393 } }
I20240501 20:01:57.601752 30073 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:57.601861 30073 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:57.601984 30073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:57.602658 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:57.596870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51377 (local address 127.8.168.126:33393)
0501 20:01:57.597240 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:57.597250 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:57.597281 (+ 31us) server_negotiation.cc:407] Connection header received
0501 20:01:57.597862 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:57.597879 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:57.598143 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:57.598494 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:57.598917 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:57.599666 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:57.600309 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:57.600592 (+ 283us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:57.601165 (+ 573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:57.601188 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:57.601201 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:57.601614 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:57.601647 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:57.601666 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:57.601791 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:57.601978 (+ 187us) server_negotiation.cc:299] Negotiation successful
0501 20:01:57.602096 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":125,"threads_started":1}
I20240501 20:01:57.602824 30073 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cf180d5c0343e7813878fb1931ef41" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33393 } }
I20240501 20:01:57.603145 30073 leader_election.cc:302] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1cf180d5c0343e7813878fb1931ef41; no voters:
I20240501 20:01:57.603394 30073 leader_election.cc:288] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:57.603490 30078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:57.603655 30078 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [term 1 LEADER]: Becoming Leader. State: Replica: e1cf180d5c0343e7813878fb1931ef41, State: Running, Role: LEADER
I20240501 20:01:57.603976 30078 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cf180d5c0343e7813878fb1931ef41" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33393 } }
I20240501 20:01:57.604532 30073 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:57.605101 30080 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1cf180d5c0343e7813878fb1931ef41. Latest consensus state: current_term: 1 leader_uuid: "e1cf180d5c0343e7813878fb1931ef41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cf180d5c0343e7813878fb1931ef41" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33393 } } }
I20240501 20:01:57.605199 30080 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:57.605441 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:01:57.605857 30079 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1cf180d5c0343e7813878fb1931ef41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cf180d5c0343e7813878fb1931ef41" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33393 } } }
I20240501 20:01:57.605954 30079 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:57.606619 30083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:57.609510 30083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:57.610937 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:57.607397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57725 (local address 127.8.168.126:33393)
0501 20:01:57.607501 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:57.607505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:57.607661 (+ 156us) server_negotiation.cc:407] Connection header received
0501 20:01:57.607791 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:57.607793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:57.607827 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:57.607892 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:57.608351 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:57.609019 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:01:57.609769 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:01:57.609944 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:01:57.610461 (+ 517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:57.610475 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:57.610479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:57.610490 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:57.610518 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:57.610530 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:57.610623 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:57.610764 (+ 141us) server_negotiation.cc:299] Negotiation successful
0501 20:01:57.610798 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240501 20:01:57.614681 30083 catalog_manager.cc:1347] Generated new cluster ID: 7bf78e0ba3a84cb19842f7edb05633ea
I20240501 20:01:57.614733 30083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:57.624811 30083 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:57.625664 30083 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:57.634307 30083 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e1cf180d5c0343e7813878fb1931ef41: Generated new TSK 0
I20240501 20:01:57.634994 30083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:57.642488 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 30005
W20240501 20:01:57.647730 29990 connection.cc:535] client connection to 127.8.168.126:33393 recv error: Network error: failed to read from TLS socket (remote: 127.8.168.126:33393): Connection reset by peer (error 104)
2024-05-01T20:01:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1237 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:57 dist-test-slave-tzct krb5kdc[30104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:57 dist-test-slave-tzct krb5kdc[30104](info): set up 2 sockets
May 01 20:01:57 dist-test-slave-tzct krb5kdc[30104](info): commencing operation
krb5kdc: starting...
W20240501 20:01:58.537142 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.875s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:58 dist-test-slave-tzct krb5kdc[30104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 20:01:58.648816 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33833
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:35463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:33833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:58.753772 30117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:58.754001 30117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:58.754073 30117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:58.757112 30117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:58.757172 30117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:58.757203 30117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:58.757231 30117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:58.757263 30117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:58.761180 30117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:35463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:33833
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:58.762006 30117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:58.762815 30117 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:58.769817 30122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:58.770268 30125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:58.770624 30123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:58.771210 30117 server_base.cc:1034] running on GCE node
I20240501 20:01:58.771667 30117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:01:58.772540 30117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:01:58.773717 30117 hybrid_clock.cc:648] HybridClock initialized: now 1714593718773703 us; error 24 us; skew 500 ppm
May 01 20:01:58 dist-test-slave-tzct krb5kdc[30104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593718, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:01:58.776300 30117 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:01:58.777256 30117 webserver.cc:460] Webserver started at http://127.8.168.126:46473/ using document root <none> and password file <none>
I20240501 20:01:58.777781 30117 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:58.777834 30117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:58.778039 30117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:58.779744 30117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "50f1994abc5d4ce9ae87f240cdbf4f89"
format_stamp: "Formatted at 2024-05-01 20:01:58 on dist-test-slave-tzct"
server_key: "95cacbf44dc22c006a088cbf86aed24d"
server_key_iv: "d896300a6a412cf13326467bc2186b8c"
server_key_version: "encryptionkey@0"
I20240501 20:01:58.780205 30117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "50f1994abc5d4ce9ae87f240cdbf4f89"
format_stamp: "Formatted at 2024-05-01 20:01:58 on dist-test-slave-tzct"
server_key: "95cacbf44dc22c006a088cbf86aed24d"
server_key_iv: "d896300a6a412cf13326467bc2186b8c"
server_key_version: "encryptionkey@0"
I20240501 20:01:58.783439 30117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 20:01:58.785661 30132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:58.786628 30117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:58.786739 30117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "50f1994abc5d4ce9ae87f240cdbf4f89"
format_stamp: "Formatted at 2024-05-01 20:01:58 on dist-test-slave-tzct"
server_key: "95cacbf44dc22c006a088cbf86aed24d"
server_key_iv: "d896300a6a412cf13326467bc2186b8c"
server_key_version: "encryptionkey@0"
I20240501 20:01:58.786829 30117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:58.791787 30117 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:58.795210 30117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:01:58.795378 30117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:58.801723 30117 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:33833
I20240501 20:01:58.801941 30184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:33833 every 8 connection(s)
I20240501 20:01:58.802647 30117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:01:58.804399 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 30117
I20240501 20:01:58.804498 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:01:58.804735 8865 external_mini_cluster.cc:1428] Setting key bfe0e1de67e8062a4022a695ac84f867
I20240501 20:01:58.805094 30185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:58.809509 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:58.805956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51769 (local address 127.8.168.126:33833)
0501 20:01:58.806346 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:58.806356 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:58.806396 (+ 40us) server_negotiation.cc:407] Connection header received
0501 20:01:58.807068 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:58.807091 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:58.807387 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:58.807765 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:58.808424 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:58.808430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:58.808442 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:58.808453 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:58.808493 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:58.808510 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:58.808650 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:58.808744 (+ 94us) server_negotiation.cc:299] Negotiation successful
0501 20:01:58.808864 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":111,"threads_started":1}
I20240501 20:01:58.811185 30185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89: Bootstrap starting.
I20240501 20:01:58.812170 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:01:58.813324 30185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:58.814088 30185 log.cc:824] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89: Log is configured to *not* fsync() on all Append() calls
I20240501 20:01:58.815857 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:01:58.814009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55125 (local address 127.8.168.126:33833)
0501 20:01:58.814117 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0501 20:01:58.814121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:01:58.814283 (+ 162us) server_negotiation.cc:407] Connection header received
0501 20:01:58.814556 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:01:58.814559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:01:58.814590 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:01:58.814652 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:01:58.815488 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:01:58.815492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:01:58.815495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:01:58.815505 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:01:58.815533 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:01:58.815545 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:01:58.815629 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0501 20:01:58.815715 (+ 86us) server_negotiation.cc:299] Negotiation successful
0501 20:01:58.815747 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240501 20:01:58.816044 30185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89: No bootstrap required, opened a new log
I20240501 20:01:58.819051 30185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f1994abc5d4ce9ae87f240cdbf4f89" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33833 } }
I20240501 20:01:58.819231 30185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:58.819280 30185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50f1994abc5d4ce9ae87f240cdbf4f89, State: Initialized, Role: FOLLOWER
I20240501 20:01:58.819658 30185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f1994abc5d4ce9ae87f240cdbf4f89" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33833 } }
I20240501 20:01:58.819773 30185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:58.819833 30185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:58.819939 30185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:58.820739 30185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f1994abc5d4ce9ae87f240cdbf4f89" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33833 } }
I20240501 20:01:58.821048 30185 leader_election.cc:302] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50f1994abc5d4ce9ae87f240cdbf4f89; no voters:
I20240501 20:01:58.821303 30185 leader_election.cc:288] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:58.821394 30195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:58.821578 30195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [term 1 LEADER]: Becoming Leader. State: Replica: 50f1994abc5d4ce9ae87f240cdbf4f89, State: Running, Role: LEADER
I20240501 20:01:58.821906 30195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f1994abc5d4ce9ae87f240cdbf4f89" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33833 } }
I20240501 20:01:58.822309 30185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:58.823307 30196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50f1994abc5d4ce9ae87f240cdbf4f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f1994abc5d4ce9ae87f240cdbf4f89" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33833 } } }
I20240501 20:01:58.823413 30196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:58.823513 30197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50f1994abc5d4ce9ae87f240cdbf4f89. Latest consensus state: current_term: 1 leader_uuid: "50f1994abc5d4ce9ae87f240cdbf4f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f1994abc5d4ce9ae87f240cdbf4f89" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33833 } } }
I20240501 20:01:58.823602 30197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89 [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:58.823746 30204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:58.826282 30204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:58.831277 30204 catalog_manager.cc:1347] Generated new cluster ID: f6553dc43ab34b038a11ca052158ebfb
I20240501 20:01:58.831338 30204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:58.839643 30204 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:58.839988 30204 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:58.849824 30204 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 50f1994abc5d4ce9ae87f240cdbf4f89: Generated new TSK 0
I20240501 20:01:58.850697 30204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:58.859907 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 30117
2024-05-01T20:01:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:01:58 dist-test-slave-tzct krb5kdc[30216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:01:58 dist-test-slave-tzct krb5kdc[30216](info): set up 2 sockets
May 01 20:01:58 dist-test-slave-tzct krb5kdc[30216](info): commencing operation
krb5kdc: starting...
W20240501 20:01:59.765094 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.887s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:01:59 dist-test-slave-tzct krb5kdc[30216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:01:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:01:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 20:01:59.875908 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:42113
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:40121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:42113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:01:59.980791 30229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:01:59.981194 30229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:01:59.981269 30229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:01:59.984447 30229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:01:59.984521 30229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:01:59.984542 30229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:01:59.984580 30229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:01:59.984597 30229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:01:59.988843 30229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:40121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:42113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:42113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:01:59.989681 30229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:01:59.990484 30229 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:59.997951 30235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:59.998263 30237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:59.998440 30234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:59.999034 30229 server_base.cc:1034] running on GCE node
I20240501 20:01:59.999519 30229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:02:00.000417 30229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:02:00.001596 30229 hybrid_clock.cc:648] HybridClock initialized: now 1714593720001580 us; error 29 us; skew 500 ppm
May 01 20:01:59 dist-test-slave-tzct krb5kdc[30216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593719, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:02:00.004211 30229 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:02:00.005146 30229 webserver.cc:460] Webserver started at http://127.8.168.126:40527/ using document root <none> and password file <none>
I20240501 20:02:00.005659 30229 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:00.005713 30229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:02:00.005882 30229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:02:00.007640 30229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "cafa161fca81430b9d0883fe2cd101e2"
format_stamp: "Formatted at 2024-05-01 20:02:00 on dist-test-slave-tzct"
server_key: "7c0abc3146e567db37bae9fef1ab2767"
server_key_iv: "cc9ca658d52f69a41cfe119dfaea52cc"
server_key_version: "encryptionkey@0"
I20240501 20:02:00.008096 30229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "cafa161fca81430b9d0883fe2cd101e2"
format_stamp: "Formatted at 2024-05-01 20:02:00 on dist-test-slave-tzct"
server_key: "7c0abc3146e567db37bae9fef1ab2767"
server_key_iv: "cc9ca658d52f69a41cfe119dfaea52cc"
server_key_version: "encryptionkey@0"
I20240501 20:02:00.011432 30229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 20:02:00.013706 30244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:00.014700 30229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:02:00.014803 30229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "cafa161fca81430b9d0883fe2cd101e2"
format_stamp: "Formatted at 2024-05-01 20:02:00 on dist-test-slave-tzct"
server_key: "7c0abc3146e567db37bae9fef1ab2767"
server_key_iv: "cc9ca658d52f69a41cfe119dfaea52cc"
server_key_version: "encryptionkey@0"
I20240501 20:02:00.014904 30229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:00.047901 30229 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:00.051049 30229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:02:00.051231 30229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:00.057947 30229 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:42113
I20240501 20:02:00.058207 30296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:42113 every 8 connection(s)
I20240501 20:02:00.059134 30229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:02:00.061497 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 30229
I20240501 20:02:00.061614 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:02:00.061707 30297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:00.061847 8865 external_mini_cluster.cc:1428] Setting key 5620961b6ccf4df11d90c3d4db810d4d
I20240501 20:02:00.068241 30297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2: Bootstrap starting.
May 01 20:02:00 dist-test-slave-tzct krb5kdc[30216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:02:00.070580 30297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:00.071444 30297 log.cc:824] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2: Log is configured to *not* fsync() on all Append() calls
I20240501 20:02:00.073470 30297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2: No bootstrap required, opened a new log
I20240501 20:02:00.074779 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:00.063070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41427 (local address 127.8.168.126:42113)
0501 20:02:00.063487 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:00.063498 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:00.063538 (+ 40us) server_negotiation.cc:407] Connection header received
0501 20:02:00.064230 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:00.064256 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:00.064636 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:00.065070 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:00.065867 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:00.067023 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:00.067725 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:00.068010 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:00.070237 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:00.070253 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:00.070267 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:00.070312 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:00.072186 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:00.072583 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:00.072589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:00.072594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:00.072686 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:00.072913 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:00.072916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:00.072918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:00.073343 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:00.073544 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:00.073770 (+ 226us) server_negotiation.cc:299] Negotiation successful
0501 20:02:00.074049 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":119,"threads_started":1}
I20240501 20:02:00.077138 30297 raft_consensus.cc:348] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafa161fca81430b9d0883fe2cd101e2" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42113 } }
I20240501 20:02:00.077328 30297 raft_consensus.cc:374] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:00.077384 30297 raft_consensus.cc:729] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cafa161fca81430b9d0883fe2cd101e2, State: Initialized, Role: FOLLOWER
I20240501 20:02:00.077535 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:02:00.077848 30297 consensus_queue.cc:260] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafa161fca81430b9d0883fe2cd101e2" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42113 } }
I20240501 20:02:00.078001 30297 raft_consensus.cc:388] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:02:00.078063 30297 raft_consensus.cc:482] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:02:00.078161 30297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:00.079284 30297 raft_consensus.cc:504] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafa161fca81430b9d0883fe2cd101e2" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42113 } }
I20240501 20:02:00.079780 30297 leader_election.cc:302] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cafa161fca81430b9d0883fe2cd101e2; no voters:
I20240501 20:02:00.080214 30297 leader_election.cc:288] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:02:00.080305 30307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:00.080518 30307 raft_consensus.cc:686] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [term 1 LEADER]: Becoming Leader. State: Replica: cafa161fca81430b9d0883fe2cd101e2, State: Running, Role: LEADER
I20240501 20:02:00.080986 30307 consensus_queue.cc:237] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafa161fca81430b9d0883fe2cd101e2" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42113 } }
I20240501 20:02:00.081457 30297 sys_catalog.cc:564] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:02:00.082576 30309 sys_catalog.cc:455] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cafa161fca81430b9d0883fe2cd101e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafa161fca81430b9d0883fe2cd101e2" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42113 } } }
I20240501 20:02:00.082708 30309 sys_catalog.cc:458] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [sys.catalog]: This master's current role is: LEADER
I20240501 20:02:00.083029 30308 sys_catalog.cc:455] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cafa161fca81430b9d0883fe2cd101e2. Latest consensus state: current_term: 1 leader_uuid: "cafa161fca81430b9d0883fe2cd101e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafa161fca81430b9d0883fe2cd101e2" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 42113 } } }
I20240501 20:02:00.083133 30308 sys_catalog.cc:458] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2 [sys.catalog]: This master's current role is: LEADER
W20240501 20:02:00.086197 30322 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:02:00.086270 30322 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:02:00.086346 30316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:02:00.087041 30316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:02:00.087927 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:00.079612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44649 (local address 127.8.168.126:42113)
0501 20:02:00.079772 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:00.079776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:00.079942 (+ 166us) server_negotiation.cc:407] Connection header received
0501 20:02:00.080067 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:00.080070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:00.080123 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:00.080216 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:00.081177 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:00.081944 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:00.082715 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:00.082905 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:00.084465 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:00.084484 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:00.084487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:00.084514 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:00.086223 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:00.086699 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:00.086702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:00.086704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:00.086754 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:00.086990 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:00.086993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:00.086995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:00.087170 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:00.087277 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:00.087643 (+ 366us) server_negotiation.cc:299] Negotiation successful
0501 20:02:00.087764 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20240501 20:02:00.092008 30316 catalog_manager.cc:1347] Generated new cluster ID: 679a557b50d24739b8829ea9141c7864
I20240501 20:02:00.092116 30316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:02:00.119609 30316 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:02:00.120946 30316 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:02:00.127630 30316 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cafa161fca81430b9d0883fe2cd101e2: Generated new TSK 0
I20240501 20:02:00.128355 30316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:02:00.134742 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 30229
2024-05-01T20:02:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1274 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:02:00 dist-test-slave-tzct krb5kdc[30328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:02:00 dist-test-slave-tzct krb5kdc[30328](info): set up 2 sockets
May 01 20:02:00 dist-test-slave-tzct krb5kdc[30328](info): commencing operation
krb5kdc: starting...
W20240501 20:02:01.037091 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.884s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:02:01 dist-test-slave-tzct krb5kdc[30328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:02:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:02:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 20:02:01.147372 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:45621
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:33481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:45621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:02:01.248937 30341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:02:01.249178 30341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:02:01.249250 30341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:02:01.252231 30341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:02:01.252298 30341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:02:01.252329 30341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:02:01.252354 30341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:02:01.252393 30341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:02:01.256467 30341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:33481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:45621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:45621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:02:01.257370 30341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:02:01.258229 30341 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:02:01.265228 30346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:01.265645 30347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:01.266112 30349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:01.266464 30341 server_base.cc:1034] running on GCE node
I20240501 20:02:01.267009 30341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:02:01.267917 30341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:02:01.269093 30341 hybrid_clock.cc:648] HybridClock initialized: now 1714593721269060 us; error 47 us; skew 500 ppm
May 01 20:02:01 dist-test-slave-tzct krb5kdc[30328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593721, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:02:01.271775 30341 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:02:01.272915 30341 webserver.cc:460] Webserver started at http://127.8.168.126:38941/ using document root <none> and password file <none>
I20240501 20:02:01.273485 30341 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:01.273540 30341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:02:01.273694 30341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:02:01.275492 30341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "76b16939fd2d4bd3a26681cfd76eba71"
format_stamp: "Formatted at 2024-05-01 20:02:01 on dist-test-slave-tzct"
server_key: "ef7b22f79d4349b41fb102b7fa95bddb"
server_key_iv: "8ee939914a5c7a115607a649cea9aefc"
server_key_version: "encryptionkey@0"
I20240501 20:02:01.275940 30341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "76b16939fd2d4bd3a26681cfd76eba71"
format_stamp: "Formatted at 2024-05-01 20:02:01 on dist-test-slave-tzct"
server_key: "ef7b22f79d4349b41fb102b7fa95bddb"
server_key_iv: "8ee939914a5c7a115607a649cea9aefc"
server_key_version: "encryptionkey@0"
I20240501 20:02:01.279268 30341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 20:02:01.281500 30356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:01.282586 30341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:02:01.282692 30341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "76b16939fd2d4bd3a26681cfd76eba71"
format_stamp: "Formatted at 2024-05-01 20:02:01 on dist-test-slave-tzct"
server_key: "ef7b22f79d4349b41fb102b7fa95bddb"
server_key_iv: "8ee939914a5c7a115607a649cea9aefc"
server_key_version: "encryptionkey@0"
I20240501 20:02:01.282786 30341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:01.296649 30341 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:01.299383 30341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:02:01.299536 30341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:01.306263 30341 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:45621
I20240501 20:02:01.306493 30408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:45621 every 8 connection(s)
I20240501 20:02:01.307255 30341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:02:01.309741 30409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:01.312285 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 30341
I20240501 20:02:01.312386 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:02:01.312696 8865 external_mini_cluster.cc:1428] Setting key c55108ddb769639e359b289dd0bf97f1
I20240501 20:02:01.315095 30409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71: Bootstrap starting.
I20240501 20:02:01.317178 30409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:01.317873 30409 log.cc:824] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71: Log is configured to *not* fsync() on all Append() calls
I20240501 20:02:01.319634 30409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71: No bootstrap required, opened a new log
I20240501 20:02:01.320466 30412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:01.314054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51863 (local address 127.8.168.126:45621)
0501 20:02:01.314482 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:01.314491 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:01.314526 (+ 35us) server_negotiation.cc:407] Connection header received
0501 20:02:01.315153 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:01.315171 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:01.315506 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:01.315843 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:01.316357 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:01.317174 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:01.317850 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:01.318128 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:01.319045 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:01.319063 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:01.319074 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:02:01.319455 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:01.319489 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:02:01.319507 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:01.319607 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:01.319763 (+ 156us) server_negotiation.cc:299] Negotiation successful
0501 20:02:01.319920 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":128,"threads_started":1}
I20240501 20:02:01.322635 30409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b16939fd2d4bd3a26681cfd76eba71" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45621 } }
I20240501 20:02:01.322832 30409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:01.322904 30409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b16939fd2d4bd3a26681cfd76eba71, State: Initialized, Role: FOLLOWER
I20240501 20:02:01.322990 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:02:01.323338 30409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b16939fd2d4bd3a26681cfd76eba71" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45621 } }
I20240501 20:02:01.323465 30409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:02:01.323525 30409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:02:01.323628 30409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:01.324461 30409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b16939fd2d4bd3a26681cfd76eba71" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45621 } }
I20240501 20:02:01.324800 30409 leader_election.cc:302] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76b16939fd2d4bd3a26681cfd76eba71; no voters:
I20240501 20:02:01.325124 30409 leader_election.cc:288] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:02:01.325222 30419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:01.325420 30419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [term 1 LEADER]: Becoming Leader. State: Replica: 76b16939fd2d4bd3a26681cfd76eba71, State: Running, Role: LEADER
I20240501 20:02:01.325974 30419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b16939fd2d4bd3a26681cfd76eba71" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45621 } }
I20240501 20:02:01.326246 30409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:02:01.327441 30420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76b16939fd2d4bd3a26681cfd76eba71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b16939fd2d4bd3a26681cfd76eba71" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45621 } } }
I20240501 20:02:01.327565 30420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [sys.catalog]: This master's current role is: LEADER
I20240501 20:02:01.327930 30421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76b16939fd2d4bd3a26681cfd76eba71. Latest consensus state: current_term: 1 leader_uuid: "76b16939fd2d4bd3a26681cfd76eba71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b16939fd2d4bd3a26681cfd76eba71" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 45621 } } }
I20240501 20:02:01.328081 30421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71 [sys.catalog]: This master's current role is: LEADER
I20240501 20:02:01.329094 30412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:01.324817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39795 (local address 127.8.168.126:45621)
0501 20:02:01.324947 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:01.324950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:01.325137 (+ 187us) server_negotiation.cc:407] Connection header received
0501 20:02:01.325276 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:01.325279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:01.325330 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:01.325399 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:01.325926 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:01.326612 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:01.327434 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:01.327632 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:01.328590 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:01.328608 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:01.328611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:02:01.328625 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:01.328656 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:02:01.328670 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:01.328739 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:01.328904 (+ 165us) server_negotiation.cc:299] Negotiation successful
0501 20:02:01.328968 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20240501 20:02:01.330950 30434 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:02:01.331012 30434 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:02:01.331089 30428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:02:01.331766 30428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:02:01.336694 30428 catalog_manager.cc:1347] Generated new cluster ID: c857f215acd24ceb8bb6603b75da2266
I20240501 20:02:01.336771 30428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:02:01.349949 30428 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:02:01.350847 30428 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:02:01.355684 30428 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 76b16939fd2d4bd3a26681cfd76eba71: Generated new TSK 0
I20240501 20:02:01.356334 30428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:02:01.366509 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 30341
2024-05-01T20:02:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:02:01 dist-test-slave-tzct krb5kdc[30440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:02:01 dist-test-slave-tzct krb5kdc[30440](info): set up 2 sockets
May 01 20:02:01 dist-test-slave-tzct krb5kdc[30440](info): commencing operation
krb5kdc: starting...
W20240501 20:02:02.281095 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.895s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:02:02 dist-test-slave-tzct krb5kdc[30440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:02:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:02:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 20:02:02.394152 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33597
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:33597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:02:02.496073 30453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:02:02.496297 30453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:02:02.496369 30453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:02:02.499445 30453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:02:02.499505 30453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:02:02.499536 30453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:02:02.499560 30453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:02:02.499591 30453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:02:02.503790 30453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:33597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:33597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:02:02.504673 30453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:02:02.505548 30453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:02:02.512846 30459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:02.513147 30461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:02.513311 30458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:02.513739 30453 server_base.cc:1034] running on GCE node
I20240501 20:02:02.514261 30453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:02:02.515146 30453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:02:02.516296 30453 hybrid_clock.cc:648] HybridClock initialized: now 1714593722516271 us; error 39 us; skew 500 ppm
May 01 20:02:02 dist-test-slave-tzct krb5kdc[30440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593722, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:02:02.518795 30453 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:02:02.519673 30453 webserver.cc:460] Webserver started at http://127.8.168.126:33295/ using document root <none> and password file <none>
I20240501 20:02:02.520195 30453 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:02.520247 30453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:02:02.520424 30453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:02:02.522198 30453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "396e7c6d2f4e4fd5bc5932c6aa000a77"
format_stamp: "Formatted at 2024-05-01 20:02:02 on dist-test-slave-tzct"
server_key: "509bde922bd691aeb94dbbe5f6670fa1"
server_key_iv: "db6b789761c5dde0921807c75360c12c"
server_key_version: "encryptionkey@0"
I20240501 20:02:02.522648 30453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "396e7c6d2f4e4fd5bc5932c6aa000a77"
format_stamp: "Formatted at 2024-05-01 20:02:02 on dist-test-slave-tzct"
server_key: "509bde922bd691aeb94dbbe5f6670fa1"
server_key_iv: "db6b789761c5dde0921807c75360c12c"
server_key_version: "encryptionkey@0"
I20240501 20:02:02.525828 30453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:02:02.527963 30468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:02.529042 30453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:02:02.529147 30453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "396e7c6d2f4e4fd5bc5932c6aa000a77"
format_stamp: "Formatted at 2024-05-01 20:02:02 on dist-test-slave-tzct"
server_key: "509bde922bd691aeb94dbbe5f6670fa1"
server_key_iv: "db6b789761c5dde0921807c75360c12c"
server_key_version: "encryptionkey@0"
I20240501 20:02:02.529229 30453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:02.533802 30453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:02.536680 30453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:02:02.536836 30453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:02.542819 30453 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:33597
I20240501 20:02:02.543051 30520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:33597 every 8 connection(s)
I20240501 20:02:02.543727 30453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:02:02.546126 30521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:02.549008 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 30453
I20240501 20:02:02.549124 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:02:02.549363 8865 external_mini_cluster.cc:1428] Setting key 7ab1f4b801fcbb84936791cfdc4d258b
I20240501 20:02:02.551697 30521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77: Bootstrap starting.
I20240501 20:02:02.554138 30521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:02.554167 30524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:02.550514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33773 (local address 127.8.168.126:33597)
0501 20:02:02.550914 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:02.550925 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:02.550965 (+ 40us) server_negotiation.cc:407] Connection header received
0501 20:02:02.551672 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:02.551697 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:02.552013 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:02.552387 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:02.553070 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:02.553076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:02.553088 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:02:02.553098 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:02.553134 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:02:02.553152 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:02.553294 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:02.553365 (+ 71us) server_negotiation.cc:299] Negotiation successful
0501 20:02:02.553485 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":111,"threads_started":1}
I20240501 20:02:02.554885 30521 log.cc:824] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77: Log is configured to *not* fsync() on all Append() calls
I20240501 20:02:02.556756 30521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77: No bootstrap required, opened a new log
I20240501 20:02:02.556774 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:02:02.560678 30524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:02.558832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43001 (local address 127.8.168.126:33597)
0501 20:02:02.558950 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:02.558954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:02.559164 (+ 210us) server_negotiation.cc:407] Connection header received
0501 20:02:02.559327 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:02.559331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:02.559365 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:02.559438 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:02.560289 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:02.560292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:02.560294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0501 20:02:02.560303 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:02.560329 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0501 20:02:02.560339 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:02.560409 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:02.560525 (+ 116us) server_negotiation.cc:299] Negotiation successful
0501 20:02:02.560554 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240501 20:02:02.560787 30521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396e7c6d2f4e4fd5bc5932c6aa000a77" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33597 } }
I20240501 20:02:02.561019 30521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:02:02.561077 30521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 396e7c6d2f4e4fd5bc5932c6aa000a77, State: Initialized, Role: FOLLOWER
I20240501 20:02:02.561545 30521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396e7c6d2f4e4fd5bc5932c6aa000a77" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33597 } }
I20240501 20:02:02.561700 30521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:02:02.561770 30521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:02:02.562040 30521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:02.562897 30521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396e7c6d2f4e4fd5bc5932c6aa000a77" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33597 } }
I20240501 20:02:02.563207 30521 leader_election.cc:302] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 396e7c6d2f4e4fd5bc5932c6aa000a77; no voters:
I20240501 20:02:02.563467 30521 leader_election.cc:288] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:02:02.563592 30531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:02.563781 30531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [term 1 LEADER]: Becoming Leader. State: Replica: 396e7c6d2f4e4fd5bc5932c6aa000a77, State: Running, Role: LEADER
I20240501 20:02:02.564339 30531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396e7c6d2f4e4fd5bc5932c6aa000a77" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33597 } }
I20240501 20:02:02.564546 30521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:02:02.565955 30532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "396e7c6d2f4e4fd5bc5932c6aa000a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396e7c6d2f4e4fd5bc5932c6aa000a77" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33597 } } }
I20240501 20:02:02.565989 30533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 396e7c6d2f4e4fd5bc5932c6aa000a77. Latest consensus state: current_term: 1 leader_uuid: "396e7c6d2f4e4fd5bc5932c6aa000a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "396e7c6d2f4e4fd5bc5932c6aa000a77" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 33597 } } }
I20240501 20:02:02.566150 30533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [sys.catalog]: This master's current role is: LEADER
I20240501 20:02:02.566147 30532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77 [sys.catalog]: This master's current role is: LEADER
I20240501 20:02:02.566498 30544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:02:02.569180 30544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:02:02.574229 30544 catalog_manager.cc:1347] Generated new cluster ID: 9fc669d2f9454d9abbf7e6e385c8d9c2
I20240501 20:02:02.574290 30544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:02:02.594873 30544 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:02:02.595173 30544 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:02:02.605346 30544 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 396e7c6d2f4e4fd5bc5932c6aa000a77: Generated new TSK 0
I20240501 20:02:02.605984 30544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:02:02.617331 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 30453
2024-05-01T20:02:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1249 ms)
[----------] 18 tests from AuthTokenIssuingTest (22346 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:02:02 dist-test-slave-tzct krb5kdc[30552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:02:02 dist-test-slave-tzct krb5kdc[30552](info): set up 2 sockets
May 01 20:02:02 dist-test-slave-tzct krb5kdc[30552](info): commencing operation
krb5kdc: starting...
W20240501 20:02:03.521209 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.886s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:02:03 dist-test-slave-tzct krb5kdc[30552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:02:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:02:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 20:02:03.629983 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:40229
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:40229,127.8.168.125:39113,127.8.168.124:38215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:02:03.734288 30565 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240501 20:02:03.734592 30565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:02:03.734642 30565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:02:03.734720 30565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:02:03.737846 30565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:02:03.737911 30565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:02:03.737937 30565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:02:03.737958 30565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:02:03.737989 30565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:02:03.742050 30565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:40229,127.8.168.125:39113,127.8.168.124:38215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:40229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:02:03.742894 30565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:02:03.743727 30565 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:02:03.751158 30565 server_base.cc:1034] running on GCE node
W20240501 20:02:03.751529 30571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:03.751937 30570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:03.751969 30573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:03.752444 30565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:02:03.753369 30565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:02:03.754521 30565 hybrid_clock.cc:648] HybridClock initialized: now 1714593723754512 us; error 24 us; skew 500 ppm
May 01 20:02:03 dist-test-slave-tzct krb5kdc[30552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593723, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:02:03.757158 30565 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:02:03.758045 30565 webserver.cc:460] Webserver started at http://127.8.168.126:38399/ using document root <none> and password file <none>
I20240501 20:02:03.758555 30565 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:03.758611 30565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:02:03.758774 30565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:02:03.760484 30565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "8a7f040443c54e92b317ac52934b51e3"
format_stamp: "Formatted at 2024-05-01 20:02:03 on dist-test-slave-tzct"
server_key: "3dcabb8febf223b7fa8ae43448a26838"
server_key_iv: "c5be5aadcf7ed7c98e5fb3af00dcb287"
server_key_version: "encryptionkey@0"
I20240501 20:02:03.760970 30565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "8a7f040443c54e92b317ac52934b51e3"
format_stamp: "Formatted at 2024-05-01 20:02:03 on dist-test-slave-tzct"
server_key: "3dcabb8febf223b7fa8ae43448a26838"
server_key_iv: "c5be5aadcf7ed7c98e5fb3af00dcb287"
server_key_version: "encryptionkey@0"
I20240501 20:02:03.764073 30565 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 20:02:03.766152 30580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:03.767099 30565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:02:03.767206 30565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "8a7f040443c54e92b317ac52934b51e3"
format_stamp: "Formatted at 2024-05-01 20:02:03 on dist-test-slave-tzct"
server_key: "3dcabb8febf223b7fa8ae43448a26838"
server_key_iv: "c5be5aadcf7ed7c98e5fb3af00dcb287"
server_key_version: "encryptionkey@0"
I20240501 20:02:03.767290 30565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:03.779603 30565 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:03.782452 30565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:02:03.782603 30565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:03.788836 30565 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:40229
I20240501 20:02:03.789070 30632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:40229 every 8 connection(s)
I20240501 20:02:03.789757 30565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:02:03.792078 30633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:03.794777 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 30565
I20240501 20:02:03.794909 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:02:03.795161 8865 external_mini_cluster.cc:1428] Setting key 17e091a5c1d8099dd0a0ce1e62884212
I20240501 20:02:03.795394 30633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.168.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.125@KRBTEST.COM" created.
May 01 20:02:03 dist-test-slave-tzct krb5kdc[30552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593723, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:02:03.809675 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:03.797558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:40229 (local address 127.0.0.1:50850)
0501 20:02:03.798014 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:03.798046 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:03.798770 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:03.799380 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:03.799385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:03.799844 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:03.800142 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:03.800153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:03.801101 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:03.801104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:03.801448 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:03.801455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:03.801654 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:03.802335 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:03.802375 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:03.805377 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:03.807207 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:03.807216 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:03.807223 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:03.807491 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:03.807778 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:03.807782 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:03.807784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:03.807887 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:03.808427 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:03.808433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:03.808597 (+ 164us) client_negotiation.cc:769] Sending connection context
0501 20:02:03.808821 (+ 224us) client_negotiation.cc:240] Negotiation successful
0501 20:02:03.809151 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":72,"spinlock_wait_cycles":33664,"thread_start_us":139,"threads_started":1}
I20240501 20:02:03.809674 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:03.797558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.8.168.126:40229)
0501 20:02:03.797914 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:03.797925 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:03.798142 (+ 217us) server_negotiation.cc:407] Connection header received
0501 20:02:03.799056 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:03.799061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:03.799207 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:03.799368 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:03.800276 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:03.801004 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:03.801552 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:03.801739 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:03.805528 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:03.805548 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:03.805557 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:03.805579 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:03.807099 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:03.807595 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:03.807600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:03.807601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:03.807684 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:03.807989 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:03.807991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:03.807992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:03.808306 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:03.808417 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:03.808861 (+ 444us) server_negotiation.cc:299] Negotiation successful
0501 20:02:03.809043 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":95,"threads_started":1}
I20240501 20:02:03.811946 30633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:02:03.812606 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:03.812359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:39113 (local address 127.0.0.1:37934)
0501 20:02:03.812481 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:03.812545 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20240501 20:02:03.812798 30584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.168.125:39113: connect: Connection refused (error 111)
I20240501 20:02:03.813552 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:03.813350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:39113 (local address 127.0.0.1:37944)
0501 20:02:03.813461 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:03.813502 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20240501 20:02:03.813684 30633 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.125:39113: Network error: Client connection negotiation failed: client connection to 127.8.168.125:39113: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.125.keytab.
Entry for principal kudu/127.8.168.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.125.keytab.
I20240501 20:02:03.820670 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.125:39113
--webserver_interface=127.8.168.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.125.keytab
--principal=kudu/127.8.168.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:40229,127.8.168.125:39113,127.8.168.124:38215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 20:02:03.862911 30633 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } attempt: 1
I20240501 20:02:03.863806 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:03.863473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:39113 (local address 127.0.0.1:37958)
0501 20:02:03.863637 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:03.863709 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20240501 20:02:03.864524 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:03.864288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:39113 (local address 127.0.0.1:37968)
0501 20:02:03.864407 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:03.864456 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20240501 20:02:03.864657 30633 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.125:39113: Network error: Client connection negotiation failed: client connection to 127.8.168.125:39113: connect: Connection refused (error 111)
W20240501 20:02:03.922430 30640 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240501 20:02:03.922672 30640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:02:03.922701 30640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:02:03.922765 30640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:02:03.925745 30640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:02:03.925796 30640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:02:03.925817 30640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:02:03.925832 30640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:02:03.925848 30640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:02:03.929975 30640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:40229,127.8.168.125:39113,127.8.168.124:38215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.125.keytab
--principal=kudu/127.8.168.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.125:39113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.168.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:02:03.930928 30640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:02:03.931746 30640 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:02:03.932883 30633 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } attempt: 2
I20240501 20:02:03.933693 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:03.933368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:39113 (local address 127.0.0.1:37978)
0501 20:02:03.933538 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:03.933604 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20240501 20:02:03.934460 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:03.934214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:39113 (local address 127.0.0.1:37984)
0501 20:02:03.934332 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:03.934393 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20240501 20:02:03.934635 30633 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.125:39113: Network error: Client connection negotiation failed: client connection to 127.8.168.125:39113: connect: Connection refused (error 111)
W20240501 20:02:03.939092 30646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:03.939491 30645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:03.940049 30648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:03.940508 30640 server_base.cc:1034] running on GCE node
I20240501 20:02:03.941004 30640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:02:03.941828 30640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:02:03.942968 30640 hybrid_clock.cc:648] HybridClock initialized: now 1714593723942950 us; error 32 us; skew 500 ppm
May 01 20:02:03 dist-test-slave-tzct krb5kdc[30552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593723, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:02:03.945498 30640 init.cc:377] Logged in from keytab as kudu/127.8.168.125@KRBTEST.COM (short username kudu)
I20240501 20:02:03.946391 30640 webserver.cc:460] Webserver started at http://127.8.168.125:33159/ using document root <none> and password file <none>
I20240501 20:02:03.946879 30640 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:03.946936 30640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:02:03.947106 30640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:02:03.948853 30640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-1/data/instance:
uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490"
format_stamp: "Formatted at 2024-05-01 20:02:03 on dist-test-slave-tzct"
server_key: "c27b01a24785d4fea61854357ba88048"
server_key_iv: "94cda6983672101de5bcee4dbbad9fc3"
server_key_version: "encryptionkey@0"
I20240501 20:02:03.949306 30640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-1/wal/instance:
uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490"
format_stamp: "Formatted at 2024-05-01 20:02:03 on dist-test-slave-tzct"
server_key: "c27b01a24785d4fea61854357ba88048"
server_key_iv: "94cda6983672101de5bcee4dbbad9fc3"
server_key_version: "encryptionkey@0"
I20240501 20:02:03.952356 30640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:02:03.954398 30655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:03.955371 30640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:02:03.955473 30640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-1/wal
uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490"
format_stamp: "Formatted at 2024-05-01 20:02:03 on dist-test-slave-tzct"
server_key: "c27b01a24785d4fea61854357ba88048"
server_key_iv: "94cda6983672101de5bcee4dbbad9fc3"
server_key_version: "encryptionkey@0"
I20240501 20:02:03.955559 30640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:03.970033 30640 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:03.972679 30640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:02:03.972865 30640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:03.979140 30640 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.125:39113
I20240501 20:02:03.979357 30707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.125:39113 every 8 connection(s)
I20240501 20:02:03.980085 30640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-1/data/info.pb
I20240501 20:02:03.982488 30708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:03.985626 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 30640
I20240501 20:02:03.985735 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-1/wal/instance
I20240501 20:02:03.985956 8865 external_mini_cluster.cc:1428] Setting key e8512b886daffed48c327e1f5182aa62
I20240501 20:02:03.985761 30708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.168.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.124@KRBTEST.COM" created.
May 01 20:02:03 dist-test-slave-tzct krb5kdc[30552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593723, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.125@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:02:03.999401 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:03.987364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.8.168.126:40229)
0501 20:02:03.987546 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:03.987551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:03.988207 (+ 656us) server_negotiation.cc:407] Connection header received
0501 20:02:03.989085 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:03.989089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:03.989146 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:03.989254 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:03.990476 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:03.991250 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:03.991850 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:03.992073 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:03.995996 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:03.996018 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:03.996020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:03.996048 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:03.997452 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:03.998002 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:03.998006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:03.998007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:03.998055 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:03.998327 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:03.998331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:03.998332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:03.998491 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:03.998575 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:03.999107 (+ 532us) server_negotiation.cc:299] Negotiation successful
0501 20:02:03.999230 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20240501 20:02:04.000000 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:03.987615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:40229 (local address 127.0.0.1:50852)
0501 20:02:03.988059 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:03.988101 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:03.988882 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:03.989388 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:03.989395 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:03.989783 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:03.990321 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:03.990334 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:03.991365 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:03.991367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:03.991736 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:03.991743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:03.991904 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:03.992477 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:03.992511 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:03.995824 (+ 3313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:03.997578 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:03.997585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:03.997597 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:03.997903 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:03.998140 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:03.998143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:03.998145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:03.998242 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:03.998599 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:03.998605 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:03.998868 (+ 263us) client_negotiation.cc:769] Sending connection context
0501 20:02:03.999048 (+ 180us) client_negotiation.cc:240] Negotiation successful
0501 20:02:03.999324 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":105,"threads_started":1}
I20240501 20:02:04.001382 30708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } has no permanent_uuid. Determining permanent_uuid...
May 01 20:02:04 dist-test-slave-tzct krb5kdc[30552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593723, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.125@KRBTEST.COM for kudu/127.8.168.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.124.keytab.
Entry for principal kudu/127.8.168.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.124.keytab.
I20240501 20:02:04.009385 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.001870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:39113 (local address 127.0.0.1:37992)
0501 20:02:04.002004 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.002016 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:04.002098 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:04.002684 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:04.002688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:04.002842 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:04.003027 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:04.003033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.003957 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.003960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:04.004326 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:04.004334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.004425 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.004963 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:04.004986 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:04.006133 (+ 1147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:04.008016 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:04.008019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:04.008021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:04.008170 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:04.008454 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:04.008458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:04.008460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:04.008502 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:04.008916 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:04.008919 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:04.009009 (+ 90us) client_negotiation.cc:769] Sending connection context
0501 20:02:04.009096 (+ 87us) client_negotiation.cc:240] Negotiation successful
0501 20:02:04.009215 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20240501 20:02:04.009476 30713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.001997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37992 (local address 127.8.168.125:39113)
0501 20:02:04.002311 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.002316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.002335 (+ 19us) server_negotiation.cc:407] Connection header received
0501 20:02:04.002374 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.002378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.002521 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.002625 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.003134 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.003845 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.004421 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.004605 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.006248 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:04.006271 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:04.006283 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:04.006307 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:04.007923 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.008309 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.008315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.008317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.008371 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.008577 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.008581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.008582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.008818 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:04.008959 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:04.009189 (+ 230us) server_negotiation.cc:299] Negotiation successful
0501 20:02:04.009313 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":138,"threads_started":1}
I20240501 20:02:04.009825 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.124:38215
--webserver_interface=127.8.168.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.124.keytab
--principal=kudu/127.8.168.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:40229,127.8.168.125:39113,127.8.168.124:38215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 20:02:04.011229 30708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:02:04.011945 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.011700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:38215 (local address 127.0.0.1:56546)
0501 20:02:04.011813 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.011875 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20240501 20:02:04.012115 30658 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
I20240501 20:02:04.012881 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.012628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:38215 (local address 127.0.0.1:56548)
0501 20:02:04.012786 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.012825 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20240501 20:02:04.013007 30708 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:38215: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
I20240501 20:02:04.025806 30633 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } attempt: 3
May 01 20:02:04 dist-test-slave-tzct krb5kdc[30552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593723, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for kudu/127.8.168.125@KRBTEST.COM
I20240501 20:02:04.034219 30713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.026488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37998 (local address 127.8.168.125:39113)
0501 20:02:04.026609 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.026613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.026628 (+ 15us) server_negotiation.cc:407] Connection header received
0501 20:02:04.026773 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.026776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.026859 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.026934 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.027477 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.027949 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.028549 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.028761 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.031034 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:04.031050 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:04.031052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:04.031073 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:04.032877 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.033223 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.033227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.033229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.033272 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.033494 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.033497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.033499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.033639 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:04.033721 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:04.033986 (+ 265us) server_negotiation.cc:299] Negotiation successful
0501 20:02:04.034087 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20240501 20:02:04.034217 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.026417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:39113 (local address 127.0.0.1:37998)
0501 20:02:04.026559 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.026575 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:04.026677 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:04.026979 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:04.026982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:04.027149 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:04.027355 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:04.027361 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.028065 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.028067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:04.028441 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:04.028449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.028550 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.029578 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:04.029591 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:04.030914 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:04.032977 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:04.032981 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:04.032983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:04.033137 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:04.033368 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:04.033371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:04.033372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:04.033416 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:04.033730 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:04.033733 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:04.033814 (+ 81us) client_negotiation.cc:769] Sending connection context
0501 20:02:04.033913 (+ 99us) client_negotiation.cc:240] Negotiation successful
0501 20:02:04.034041 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20240501 20:02:04.035008 30633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:02:04.035679 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.035450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:38215 (local address 127.0.0.1:56564)
0501 20:02:04.035569 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.035617 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20240501 20:02:04.036388 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.036162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:38215 (local address 127.0.0.1:56568)
0501 20:02:04.036282 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.036335 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20240501 20:02:04.036500 30633 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:38215: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
I20240501 20:02:04.062198 30708 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } attempt: 1
I20240501 20:02:04.063014 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.062711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:38215 (local address 127.0.0.1:56572)
0501 20:02:04.062860 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.062918 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20240501 20:02:04.063704 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.063501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:38215 (local address 127.0.0.1:56582)
0501 20:02:04.063606 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.063647 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20240501 20:02:04.063901 30708 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:38215: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
I20240501 20:02:04.067641 30633 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } attempt: 1
I20240501 20:02:04.068289 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.068052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:38215 (local address 127.0.0.1:56590)
0501 20:02:04.068173 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.068220 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20240501 20:02:04.068955 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.068755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:38215 (local address 127.0.0.1:56604)
0501 20:02:04.068855 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.068896 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20240501 20:02:04.069067 30633 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:38215: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
W20240501 20:02:04.116117 30714 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240501 20:02:04.116375 30714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:02:04.116423 30714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:02:04.116501 30714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:02:04.119514 30714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:02:04.119573 30714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:02:04.119602 30714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:02:04.119626 30714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:02:04.119663 30714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:02:04.123661 30714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:40229,127.8.168.125:39113,127.8.168.124:38215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.124.keytab
--principal=kudu/127.8.168.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.124:38215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.168.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:02:04.124502 30714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:02:04.125358 30714 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:02:04.132067 30708 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } attempt: 2
W20240501 20:02:04.132481 30723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:04.132874 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.132534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:38215 (local address 127.0.0.1:56620)
0501 20:02:04.132702 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.132782 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20240501 20:02:04.132889 30721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:04.132975 30720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:04.133494 30714 server_base.cc:1034] running on GCE node
I20240501 20:02:04.133594 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.133374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:38215 (local address 127.0.0.1:56630)
0501 20:02:04.133493 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.133535 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20240501 20:02:04.133752 30708 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:38215: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
I20240501 20:02:04.134002 30714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:02:04.134891 30714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:02:04.136054 30714 hybrid_clock.cc:648] HybridClock initialized: now 1714593724136040 us; error 26 us; skew 500 ppm
May 01 20:02:04 dist-test-slave-tzct krb5kdc[30552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593724, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:02:04.138597 30714 init.cc:377] Logged in from keytab as kudu/127.8.168.124@KRBTEST.COM (short username kudu)
I20240501 20:02:04.139448 30714 webserver.cc:460] Webserver started at http://127.8.168.124:34889/ using document root <none> and password file <none>
I20240501 20:02:04.139915 30714 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:04.139963 30714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:02:04.140115 30714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:02:04.141809 30714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-2/data/instance:
uuid: "575060c143614f59a3f4f880e4a65ea9"
format_stamp: "Formatted at 2024-05-01 20:02:04 on dist-test-slave-tzct"
server_key: "5e59d9c5f7a63f09764baf857ff64e85"
server_key_iv: "7edd9afed86fd4addb5476b6763a61a5"
server_key_version: "encryptionkey@0"
I20240501 20:02:04.142237 30714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-2/wal/instance:
uuid: "575060c143614f59a3f4f880e4a65ea9"
format_stamp: "Formatted at 2024-05-01 20:02:04 on dist-test-slave-tzct"
server_key: "5e59d9c5f7a63f09764baf857ff64e85"
server_key_iv: "7edd9afed86fd4addb5476b6763a61a5"
server_key_version: "encryptionkey@0"
I20240501 20:02:04.144208 30633 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } attempt: 2
I20240501 20:02:04.144907 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.144623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:38215 (local address 127.0.0.1:56644)
0501 20:02:04.144777 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.144828 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20240501 20:02:04.145404 30714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:02:04.145579 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.145354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:38215 (local address 127.0.0.1:56646)
0501 20:02:04.145467 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.145510 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20240501 20:02:04.145686 30633 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:38215: Network error: Client connection negotiation failed: client connection to 127.8.168.124:38215: connect: Connection refused (error 111)
I20240501 20:02:04.147562 30730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:04.148523 30714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:02:04.148627 30714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-2/wal
uuid: "575060c143614f59a3f4f880e4a65ea9"
format_stamp: "Formatted at 2024-05-01 20:02:04 on dist-test-slave-tzct"
server_key: "5e59d9c5f7a63f09764baf857ff64e85"
server_key_iv: "7edd9afed86fd4addb5476b6763a61a5"
server_key_version: "encryptionkey@0"
I20240501 20:02:04.148710 30714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:04.163513 30714 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:04.166467 30714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:02:04.166631 30714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:04.173081 30714 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.124:38215
I20240501 20:02:04.173296 30782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.124:38215 every 8 connection(s)
I20240501 20:02:04.174001 30714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-2/data/info.pb
I20240501 20:02:04.175595 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 30714
I20240501 20:02:04.175706 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1714593353892612-8865-0/minicluster-data/master-2/wal/instance
I20240501 20:02:04.175927 8865 external_mini_cluster.cc:1428] Setting key 7473f3efdd8c15235c6185af55dc64af
I20240501 20:02:04.176441 30783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:04.180398 30783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } has no permanent_uuid. Determining permanent_uuid...
May 01 20:02:04 dist-test-slave-tzct krb5kdc[30552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.124@KRBTEST.COM
May 01 20:02:04 dist-test-slave-tzct krb5kdc[30552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593724, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.124@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:02:04.187970 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.177031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56662 (local address 127.8.168.124:38215)
0501 20:02:04.177423 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.177435 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.177474 (+ 39us) server_negotiation.cc:407] Connection header received
0501 20:02:04.178207 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.178231 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.178577 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.178962 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.179705 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.180836 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.181466 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.181728 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.183874 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:04.183926 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:04.183933 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:04.183964 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:04.185706 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.186122 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.186126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.186130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.186188 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.186409 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.186413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.186415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.186735 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:04.186901 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:04.187185 (+ 284us) server_negotiation.cc:299] Negotiation successful
0501 20:02:04.187402 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":105,"threads_started":1}
I20240501 20:02:04.189641 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.181912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:40229 (local address 127.0.0.1:50864)
0501 20:02:04.182232 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.182265 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:04.182373 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:04.182649 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:04.182655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:04.183008 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:04.183212 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:04.183218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.184099 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.184102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:04.184571 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:04.184581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.184678 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.185197 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:04.185209 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:04.186570 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:04.188209 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:04.188215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:04.188219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:04.188466 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:04.188675 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:04.188677 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:04.188679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:04.188754 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:04.189101 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:04.189106 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:04.189208 (+ 102us) client_negotiation.cc:769] Sending connection context
0501 20:02:04.189344 (+ 136us) client_negotiation.cc:240] Negotiation successful
0501 20:02:04.189484 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":94,"threads_started":1}
I20240501 20:02:04.189648 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.182007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.8.168.126:40229)
0501 20:02:04.182137 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.182141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.182338 (+ 197us) server_negotiation.cc:407] Connection header received
0501 20:02:04.182466 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.182470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.182517 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.182584 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.183339 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.183991 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.184685 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.184895 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.186691 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:04.186708 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:04.186710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:04.186733 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:04.188082 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.188551 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.188554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.188555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.188595 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.188846 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.188849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.188850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.189001 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:04.189079 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:04.189380 (+ 301us) server_negotiation.cc:299] Negotiation successful
0501 20:02:04.189490 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20240501 20:02:04.190816 30783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } has no permanent_uuid. Determining permanent_uuid...
May 01 20:02:04 dist-test-slave-tzct krb5kdc[30552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
May 01 20:02:04 dist-test-slave-tzct krb5kdc[30552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593724, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.124@KRBTEST.COM for kudu/127.8.168.125@KRBTEST.COM
I20240501 20:02:04.198000 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.190131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50878 (local address 127.8.168.126:40229)
0501 20:02:04.190249 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.190252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.190297 (+ 45us) server_negotiation.cc:407] Connection header received
0501 20:02:04.190418 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.190421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.190464 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.190538 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.191303 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.191899 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.192536 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.192713 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.194795 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:04.194816 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:04.194819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:04.194845 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:04.196457 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.196866 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.196870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.196872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.196918 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.197126 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.197129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.197131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.197302 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:04.197428 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:04.197623 (+ 195us) server_negotiation.cc:299] Negotiation successful
0501 20:02:04.197861 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20240501 20:02:04.198000 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.191252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:39113 (local address 127.0.0.1:38006)
0501 20:02:04.191397 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.191410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:04.191483 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:04.191735 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:04.191738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:04.191852 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:04.192015 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:04.192022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.192756 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.192759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:04.193157 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:04.193166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.193252 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.193639 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:04.193651 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:04.195040 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:04.196678 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:04.196680 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:04.196682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:04.196908 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:04.197144 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:04.197147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:04.197149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:04.197194 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:04.197515 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:04.197517 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:04.197615 (+ 98us) client_negotiation.cc:769] Sending connection context
0501 20:02:04.197719 (+ 104us) client_negotiation.cc:240] Negotiation successful
0501 20:02:04.197833 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20240501 20:02:04.198009 30713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.191381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38006 (local address 127.8.168.125:39113)
0501 20:02:04.191524 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.191528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.191543 (+ 15us) server_negotiation.cc:407] Connection header received
0501 20:02:04.191583 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.191586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.191636 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.191716 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.192131 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.192620 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.193279 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.193477 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.195173 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:04.195191 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:04.195194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:04.195216 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:04.196562 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.197011 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.197014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.197016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.197063 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.197270 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.197273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.197274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.197418 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:04.197511 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:04.197791 (+ 280us) server_negotiation.cc:299] Negotiation successful
0501 20:02:04.197878 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20240501 20:02:04.198870 30783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } has no permanent_uuid. Determining permanent_uuid...
May 01 20:02:04 dist-test-slave-tzct krb5kdc[30552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593724, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.124@KRBTEST.COM for kudu/127.8.168.124@KRBTEST.COM
May 01 20:02:04 dist-test-slave-tzct krb5kdc[30552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.125@KRBTEST.COM
I20240501 20:02:04.205827 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.199292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:38215 (local address 127.0.0.1:56672)
0501 20:02:04.199416 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.199530 (+ 114us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:04.199588 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:04.199800 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:04.199802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:04.199927 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:04.200071 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:04.200077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.201244 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.201246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:04.201651 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:04.201661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.201752 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.202134 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:04.202146 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:04.203185 (+ 1039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:04.204688 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:04.204691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:04.204694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:04.204877 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:04.205083 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:04.205086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:04.205088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:04.205133 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:04.205408 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:04.205411 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:04.205486 (+ 75us) client_negotiation.cc:769] Sending connection context
0501 20:02:04.205565 (+ 79us) client_negotiation.cc:240] Negotiation successful
0501 20:02:04.205667 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20240501 20:02:04.205847 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.199478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.8.168.124:38215)
0501 20:02:04.199579 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.199583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.199594 (+ 11us) server_negotiation.cc:407] Connection header received
0501 20:02:04.199667 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.199670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.199711 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.199783 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.200177 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.201131 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.201762 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.201962 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.203296 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:04.203311 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:04.203315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:04.203338 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:04.204585 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.204959 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.204962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.204964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.205008 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.205204 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.205207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.205209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.205324 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:04.205403 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:04.205621 (+ 218us) server_negotiation.cc:299] Negotiation successful
0501 20:02:04.205731 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":40}
I20240501 20:02:04.207077 30713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.199211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38012 (local address 127.8.168.125:39113)
0501 20:02:04.199315 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.199318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.199348 (+ 30us) server_negotiation.cc:407] Connection header received
0501 20:02:04.199456 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.199459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.199502 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.199581 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.200254 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.200671 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.201390 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.201579 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.203886 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:04.203900 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:04.203902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:04.204005 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:04.205268 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.205611 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.205614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.205615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.205653 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.205847 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.205850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.205852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.206028 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:04.206528 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:04.206819 (+ 291us) server_negotiation.cc:299] Negotiation successful
0501 20:02:04.206926 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240501 20:02:04.208781 30783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9: Bootstrap starting.
I20240501 20:02:04.210211 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:02:04.211205 30783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:04.211897 30783 log.cc:824] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9: Log is configured to *not* fsync() on all Append() calls
I20240501 20:02:04.211370 30607 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8a7f040443c54e92b317ac52934b51e3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50878
I20240501 20:02:04.213627 30783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9: No bootstrap required, opened a new log
I20240501 20:02:04.216028 30783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } }
I20240501 20:02:04.216181 30783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575060c143614f59a3f4f880e4a65ea9, State: Initialized, Role: FOLLOWER
I20240501 20:02:04.216457 30783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } }
I20240501 20:02:04.216881 30789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } } }
I20240501 20:02:04.217000 30789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:02:04.217586 30783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9 [sys.catalog]: configured and running, proceeding with master startup.
W20240501 20:02:04.221905 30800 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:02:04.221995 30800 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:02:04.220992 30714 master_runner.cc:186] Error getting master registration for 127.8.168.126:40229: OK, instance_id { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" instance_seqno: 1714593723786935 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240501 20:02:04.223429 30714 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:02:04.224875 30708 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } attempt: 3
May 01 20:02:04 dist-test-slave-tzct krb5kdc[30552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593723, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.125@KRBTEST.COM for kudu/127.8.168.124@KRBTEST.COM
I20240501 20:02:04.232545 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.225364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.8.168.124:38215)
0501 20:02:04.225490 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.225495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.225511 (+ 16us) server_negotiation.cc:407] Connection header received
0501 20:02:04.225618 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.225621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.225670 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.225753 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.226331 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.227038 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.227776 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.227967 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.229862 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:04.229876 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:04.229878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:04.229900 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:04.231201 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.231573 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.231576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.231577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.231628 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.231836 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.231839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.231841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.231985 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:04.232074 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:04.232312 (+ 238us) server_negotiation.cc:299] Negotiation successful
0501 20:02:04.232408 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20240501 20:02:04.232545 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.225298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:38215 (local address 127.0.0.1:56678)
0501 20:02:04.225422 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.225437 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:04.225521 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:04.225777 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:04.225780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:04.225984 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:04.226221 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:04.226228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.227138 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.227141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:04.227670 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:04.227679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.227775 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.228319 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:04.228335 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:04.229725 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:04.231311 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:04.231315 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:04.231318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:04.231483 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:04.231711 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:04.231714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:04.231716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:04.231759 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:04.232074 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:04.232077 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:04.232155 (+ 78us) client_negotiation.cc:769] Sending connection context
0501 20:02:04.232246 (+ 91us) client_negotiation.cc:240] Negotiation successful
0501 20:02:04.232367 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20240501 20:02:04.236143 30708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490: Bootstrap starting.
I20240501 20:02:04.238710 30708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:04.239482 30708 log.cc:824] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490: Log is configured to *not* fsync() on all Append() calls
I20240501 20:02:04.241974 30708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490: No bootstrap required, opened a new log
I20240501 20:02:04.244678 30708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } }
I20240501 20:02:04.244836 30633 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } attempt: 3
I20240501 20:02:04.245007 30708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ff6782c8c3e4e6197a2a6e3e8d6a490, State: Initialized, Role: FOLLOWER
I20240501 20:02:04.245330 30708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } }
I20240501 20:02:04.245769 30803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } } }
I20240501 20:02:04.245940 30803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:02:04.246462 30708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490 [sys.catalog]: configured and running, proceeding with master startup.
May 01 20:02:04 dist-test-slave-tzct krb5kdc[30552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593723, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for kudu/127.8.168.124@KRBTEST.COM
W20240501 20:02:04.251349 30814 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:02:04.251457 30814 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:02:04.251533 30640 master_runner.cc:186] Error getting master registration for 127.8.168.126:40229: OK, instance_id { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" instance_seqno: 1714593723786935 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240501 20:02:04.254477 30640 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:02:04.254627 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.245321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:38215 (local address 127.0.0.1:56680)
0501 20:02:04.245612 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:04.245627 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:04.245723 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:04.246346 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:04.246349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:04.246553 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:04.246797 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:04.246804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.247797 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.247800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:04.248358 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:04.248368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.248481 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.249216 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:04.249234 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:04.251032 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:04.253084 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:04.253088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:04.253090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:04.253314 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:04.253581 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:04.253584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:04.253587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:04.253650 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:04.254110 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:04.254113 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:04.254208 (+ 95us) client_negotiation.cc:769] Sending connection context
0501 20:02:04.254311 (+ 103us) client_negotiation.cc:240] Negotiation successful
0501 20:02:04.254439 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20240501 20:02:04.255120 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.245866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.8.168.124:38215)
0501 20:02:04.246037 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.246042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.246055 (+ 13us) server_negotiation.cc:407] Connection header received
0501 20:02:04.246096 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.246099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.246145 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.246218 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.246944 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.247662 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.248490 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.248676 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.251143 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:04.251163 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:04.251166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:04.251193 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:04.252935 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.253408 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.253411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.253413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.253479 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.253764 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.253768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.253770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.253971 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:04.254065 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:04.254833 (+ 768us) server_negotiation.cc:299] Negotiation successful
0501 20:02:04.254964 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20240501 20:02:04.258147 30633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3: Bootstrap starting.
I20240501 20:02:04.260682 30633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:04.261540 30633 log.cc:824] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3: Log is configured to *not* fsync() on all Append() calls
I20240501 20:02:04.263617 30633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3: No bootstrap required, opened a new log
I20240501 20:02:04.266295 30633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } }
I20240501 20:02:04.266494 30633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a7f040443c54e92b317ac52934b51e3, State: Initialized, Role: FOLLOWER
I20240501 20:02:04.266832 30633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } }
I20240501 20:02:04.267364 30817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } } }
I20240501 20:02:04.267606 30817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:02:04.268034 30633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:02:04.268085 30607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:02:04.268173 30607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:04.269822 30607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } }
I20240501 20:02:04.270813 30607 leader_election.cc:288] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ff6782c8c3e4e6197a2a6e3e8d6a490 (127.8.168.125:39113), 575060c143614f59a3f4f880e4a65ea9 (127.8.168.124:38215)
I20240501 20:02:04.271714 30682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a7f040443c54e92b317ac52934b51e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490"
I20240501 20:02:04.271950 30682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:04.273234 30682 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a7f040443c54e92b317ac52934b51e3 in term 1.
I20240501 20:02:04.273694 30757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a7f040443c54e92b317ac52934b51e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "575060c143614f59a3f4f880e4a65ea9"
I20240501 20:02:04.273903 30757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:04.275097 30757 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a7f040443c54e92b317ac52934b51e3 in term 1.
W20240501 20:02:04.275712 30828 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:02:04.275825 30828 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:02:04.277254 30583 leader_election.cc:302] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 575060c143614f59a3f4f880e4a65ea9, 8a7f040443c54e92b317ac52934b51e3; no voters:
I20240501 20:02:04.277626 30817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:04.277832 30817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [term 1 LEADER]: Becoming Leader. State: Replica: 8a7f040443c54e92b317ac52934b51e3, State: Running, Role: LEADER
I20240501 20:02:04.278151 30817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } }
I20240501 20:02:04.281203 30835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a7f040443c54e92b317ac52934b51e3. Latest consensus state: current_term: 1 leader_uuid: "8a7f040443c54e92b317ac52934b51e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } } }
I20240501 20:02:04.281401 30835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [sys.catalog]: This master's current role is: LEADER
W20240501 20:02:04.283923 30586 tablet.cc:2369] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:02:04.292829 30713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.277040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.8.168.125:39113)
0501 20:02:04.277159 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.277164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.284542 (+ 7378us) server_negotiation.cc:407] Connection header received
0501 20:02:04.284753 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.284756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.284807 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.284861 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.285977 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.286702 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.287527 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.287719 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.289350 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:04.289371 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:04.289375 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:04.289402 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:04.291212 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.291654 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.291658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.291660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.291709 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.291942 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.291946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.291948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.292115 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:04.292211 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:04.292536 (+ 325us) server_negotiation.cc:299] Negotiation successful
0501 20:02:04.292652 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20240501 20:02:04.293372 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.284035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.8.168.124:38215)
0501 20:02:04.284249 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.284253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.285621 (+ 1368us) server_negotiation.cc:407] Connection header received
0501 20:02:04.285784 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.285788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.285832 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.285879 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.286897 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.287591 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.288414 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.288601 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.290054 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:04.290070 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:04.290073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:04.290099 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:04.291797 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.292223 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.292226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.292228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.292275 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.292512 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.292515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.292518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.292675 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:04.292792 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:04.293118 (+ 326us) server_negotiation.cc:299] Negotiation successful
0501 20:02:04.293222 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20240501 20:02:04.297124 30837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:02:04.298206 30837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:02:04.301738 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:04.284001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.8.168.126:40229)
0501 20:02:04.284109 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.284113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.294181 (+ 10068us) server_negotiation.cc:407] Connection header received
0501 20:02:04.294342 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.294345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.294399 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.294447 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.295389 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.296090 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:04.296900 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:04.297102 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:04.298515 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:04.298532 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:04.298536 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:04.298563 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:04.300136 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.300554 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.300558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.300560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.300607 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:04.300866 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:04.300869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:04.300871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:04.301057 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:04.301161 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:04.301478 (+ 317us) server_negotiation.cc:299] Negotiation successful
0501 20:02:04.301586 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240501 20:02:04.316884 30757 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9 [term 1 FOLLOWER]: Refusing update from remote peer 8a7f040443c54e92b317ac52934b51e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:02:04.317634 30835 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:02:04.321619 30682 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490 [term 1 FOLLOWER]: Refusing update from remote peer 8a7f040443c54e92b317ac52934b51e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:02:04.321789 30789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a7f040443c54e92b317ac52934b51e3. Latest consensus state: current_term: 1 leader_uuid: "8a7f040443c54e92b317ac52934b51e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } } }
I20240501 20:02:04.321887 30789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:02:04.322214 30835 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:02:04.327195 30803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a7f040443c54e92b317ac52934b51e3. Latest consensus state: current_term: 1 leader_uuid: "8a7f040443c54e92b317ac52934b51e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } } }
I20240501 20:02:04.327313 30803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:02:04.334806 30835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a7f040443c54e92b317ac52934b51e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } } }
I20240501 20:02:04.334944 30835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [sys.catalog]: This master's current role is: LEADER
I20240501 20:02:04.335041 30835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a7f040443c54e92b317ac52934b51e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } } }
I20240501 20:02:04.335100 30835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3 [sys.catalog]: This master's current role is: LEADER
I20240501 20:02:04.335311 30840 mvcc.cc:205] Tried to move back new op lower bound from 7022975894784610304 to 7022975894646624256. Current Snapshot: MvccSnapshot[applied={T|T < 7022975894784610304}]
I20240501 20:02:04.336335 30789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a7f040443c54e92b317ac52934b51e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } } }
I20240501 20:02:04.336452 30789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:02:04.337396 30803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a7f040443c54e92b317ac52934b51e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a7f040443c54e92b317ac52934b51e3" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 40229 } } peers { permanent_uuid: "9ff6782c8c3e4e6197a2a6e3e8d6a490" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 39113 } } peers { permanent_uuid: "575060c143614f59a3f4f880e4a65ea9" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 38215 } } }
I20240501 20:02:04.337498 30803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:02:04.338446 30841 mvcc.cc:205] Tried to move back new op lower bound from 7022975894784610304 to 7022975894646624256. Current Snapshot: MvccSnapshot[applied={T|T < 7022975894784610304}]
I20240501 20:02:04.338874 30843 mvcc.cc:205] Tried to move back new op lower bound from 7022975894784610304 to 7022975894646624256. Current Snapshot: MvccSnapshot[applied={T|T < 7022975894784610304}]
I20240501 20:02:04.343469 30837 catalog_manager.cc:1347] Generated new cluster ID: 8b5d785ab4604a6fa88ace9be2b5195a
I20240501 20:02:04.343531 30837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:02:04.365106 30837 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:02:04.366458 30837 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:02:04.381856 30837 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8a7f040443c54e92b317ac52934b51e3: Generated new TSK 0
I20240501 20:02:04.382701 30837 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240501 20:02:04.415920 30636 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:04.412828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50886 (local address 127.8.168.126:40229)
0501 20:02:04.412974 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.412979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.414817 (+ 1838us) server_negotiation.cc:407] Connection header received
0501 20:02:04.415039 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.415043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.415101 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.415155 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.415772 (+ 617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50886: BlockingRecv error: recv got EOF from 127.0.0.1:50886 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20240501 20:02:04.417304 30786 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:04.413299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.8.168.124:38215)
0501 20:02:04.413414 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.413418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.416402 (+ 2984us) server_negotiation.cc:407] Connection header received
0501 20:02:04.416563 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.416566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.416615 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.416661 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.417184 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56684: BlockingRecv error: recv got EOF from 127.0.0.1:56684 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20240501 20:02:04.418596 30713 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:04.413716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38032 (local address 127.8.168.125:39113)
0501 20:02:04.413829 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.413833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.417750 (+ 3917us) server_negotiation.cc:407] Connection header received
0501 20:02:04.417923 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.417927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.417977 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.418028 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.418485 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38032: BlockingRecv error: recv got EOF from 127.0.0.1:38032 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20240501 20:02:04.422950 30713 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:04.421511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38048 (local address 127.8.168.125:39113)
0501 20:02:04.421616 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.421620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.422035 (+ 415us) server_negotiation.cc:407] Connection header received
0501 20:02:04.422246 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.422250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.422320 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.422375 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.422835 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38048: BlockingRecv error: recv got EOF from 127.0.0.1:38048 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20240501 20:02:04.427615 30713 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:04.426209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38052 (local address 127.8.168.125:39113)
0501 20:02:04.426308 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.426312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.426771 (+ 459us) server_negotiation.cc:407] Connection header received
0501 20:02:04.426964 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.426968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.427018 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.427066 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.427516 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38052: BlockingRecv error: recv got EOF from 127.0.0.1:38052 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20240501 20:02:04.432873 30713 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:04.431430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38054 (local address 127.8.168.125:39113)
0501 20:02:04.431530 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.431534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.432013 (+ 479us) server_negotiation.cc:407] Connection header received
0501 20:02:04.432189 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.432193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.432243 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.432292 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.432765 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38054: BlockingRecv error: recv got EOF from 127.0.0.1:38054 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20240501 20:02:04.440178 30713 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:04.438700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.8.168.125:39113)
0501 20:02:04.438824 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.438828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.439310 (+ 482us) server_negotiation.cc:407] Connection header received
0501 20:02:04.439488 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.439491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.439542 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.439588 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.440077 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38062: BlockingRecv error: recv got EOF from 127.0.0.1:38062 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20240501 20:02:04.452095 30713 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:04.450319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38072 (local address 127.8.168.125:39113)
0501 20:02:04.450444 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.450448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.451022 (+ 574us) server_negotiation.cc:407] Connection header received
0501 20:02:04.451231 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.451235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.451293 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.451348 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.451980 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38072: BlockingRecv error: recv got EOF from 127.0.0.1:38072 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20240501 20:02:04.472004 30713 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:04.470611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38078 (local address 127.8.168.125:39113)
0501 20:02:04.470761 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.470766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.470916 (+ 150us) server_negotiation.cc:407] Connection header received
0501 20:02:04.471135 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.471139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.471207 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.471318 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.471880 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38078: BlockingRecv error: recv got EOF from 127.0.0.1:38078 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20240501 20:02:04.506831 30713 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:04.505877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38086 (local address 127.8.168.125:39113)
0501 20:02:04.506011 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.506015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.506117 (+ 102us) server_negotiation.cc:407] Connection header received
0501 20:02:04.506240 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.506242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.506292 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.506375 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.506731 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38086: BlockingRecv error: recv got EOF from 127.0.0.1:38086 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20240501 20:02:04.573992 30713 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:04.572886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.8.168.125:39113)
0501 20:02:04.573012 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.573015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.573105 (+ 90us) server_negotiation.cc:407] Connection header received
0501 20:02:04.573235 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.573238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.573297 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.573387 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.573867 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38094: BlockingRecv error: recv got EOF from 127.0.0.1:38094 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20240501 20:02:04.705121 30713 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:04.703913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38104 (local address 127.8.168.125:39113)
0501 20:02:04.704103 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.704109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.704241 (+ 132us) server_negotiation.cc:407] Connection header received
0501 20:02:04.704382 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.704385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.704443 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.704517 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.704997 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38104: BlockingRecv error: recv got EOF from 127.0.0.1:38104 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20240501 20:02:04.964116 30713 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:04.962995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38120 (local address 127.8.168.125:39113)
0501 20:02:04.963145 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:04.963149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:04.963248 (+ 99us) server_negotiation.cc:407] Connection header received
0501 20:02:04.963397 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:04.963400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:04.963464 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:04.963534 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:04.963997 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38120: BlockingRecv error: recv got EOF from 127.0.0.1:38120 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20240501 20:02:05.223212 30800 catalog_manager.cc:1259] Loaded cluster ID: 8b5d785ab4604a6fa88ace9be2b5195a
I20240501 20:02:05.223296 30800 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9: loading cluster ID for follower catalog manager: success
I20240501 20:02:05.224954 30800 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9: acquiring CA information for follower catalog manager: success
I20240501 20:02:05.226078 30800 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 575060c143614f59a3f4f880e4a65ea9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 20:02:05.252887 30814 catalog_manager.cc:1259] Loaded cluster ID: 8b5d785ab4604a6fa88ace9be2b5195a
I20240501 20:02:05.252975 30814 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490: loading cluster ID for follower catalog manager: success
I20240501 20:02:05.254688 30814 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490: acquiring CA information for follower catalog manager: success
I20240501 20:02:05.255827 30814 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9ff6782c8c3e4e6197a2a6e3e8d6a490: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 20:02:05.481429 30913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:05.478146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38126 (local address 127.8.168.125:39113)
0501 20:02:05.478350 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:05.478354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:05.478389 (+ 35us) server_negotiation.cc:407] Connection header received
0501 20:02:05.478531 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:05.478533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:05.478651 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:05.478760 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:02:05.479190 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:05.479689 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:05.480327 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:05.480459 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:05.480551 (+ 92us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:02:05.481091 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:02:05.481186 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:05.481258 (+ 72us) server_negotiation.cc:299] Negotiation successful
0501 20:02:05.481306 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":63,"threads_started":1}
I20240501 20:02:05.487293 30919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:05.484283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56688 (local address 127.8.168.124:38215)
0501 20:02:05.484526 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:05.484529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:05.484544 (+ 15us) server_negotiation.cc:407] Connection header received
0501 20:02:05.484637 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:05.484639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:05.484790 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:05.484929 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:02:05.485199 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:05.485693 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:05.486259 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:05.486400 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:05.486483 (+ 83us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:02:05.486925 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:02:05.487017 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:05.487084 (+ 67us) server_negotiation.cc:299] Negotiation successful
0501 20:02:05.487130 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
I20240501 20:02:05.493646 30919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:05.490153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.8.168.124:38215)
0501 20:02:05.490273 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:05.490277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:05.490852 (+ 575us) server_negotiation.cc:407] Connection header received
0501 20:02:05.490926 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:05.490929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:05.490990 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:05.491054 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:02:05.491474 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:05.492164 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:05.492822 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:05.493005 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:05.493215 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:02:05.493290 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:02:05.493359 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:05.493501 (+ 142us) server_negotiation.cc:299] Negotiation successful
0501 20:02:05.493547 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240501 20:02:05.493631 30913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:05.490120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.8.168.125:39113)
0501 20:02:05.490392 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:05.490396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:05.490625 (+ 229us) server_negotiation.cc:407] Connection header received
0501 20:02:05.490746 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:05.490750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:05.490811 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:05.490876 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:02:05.491319 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:05.492047 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:05.492783 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:05.492962 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:05.493152 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:02:05.493252 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:02:05.493328 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:05.493435 (+ 107us) server_negotiation.cc:299] Negotiation successful
0501 20:02:05.493493 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20240501 20:02:05.494115 30924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:05.490094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50894 (local address 127.8.168.126:40229)
0501 20:02:05.490325 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:05.490329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:05.490588 (+ 259us) server_negotiation.cc:407] Connection header received
0501 20:02:05.490753 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:05.490756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:05.490900 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:05.491008 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:02:05.491366 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:05.491904 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:05.492779 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:05.493005 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:05.493121 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:02:05.493483 (+ 362us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:02:05.493571 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:05.493692 (+ 121us) server_negotiation.cc:299] Negotiation successful
0501 20:02:05.493979 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
I20240501 20:02:05.497695 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 30565
I20240501 20:02:05.503067 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 30640
I20240501 20:02:05.507966 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 30714
2024-05-01T20:02:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2891 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 20:02:05 dist-test-slave-tzct krb5kdc[30930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 20:02:05 dist-test-slave-tzct krb5kdc[30930](info): set up 2 sockets
May 01 20:02:05 dist-test-slave-tzct krb5kdc[30930](info): commencing operation
krb5kdc: starting...
W20240501 20:02:06.409173 8865 mini_kdc.cc:121] Time spent starting KDC: real 0.882s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/test-user.keytab.
May 01 20:02:06 dist-test-slave-tzct krb5kdc[30930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-01T20:02:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:02:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.168.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
Entry for principal kudu/127.8.168.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab.
I20240501 20:02:06.521755 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:32929
--webserver_interface=127.8.168.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:32929,127.8.168.125:36835,127.8.168.124:43935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240501 20:02:06.629416 30943 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240501 20:02:06.629674 30943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:02:06.629706 30943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:02:06.629766 30943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:02:06.632975 30943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:02:06.633044 30943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:02:06.633066 30943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:02:06.633082 30943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:02:06.633101 30943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:02:06.637445 30943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:32929,127.8.168.125:36835,127.8.168.124:43935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.126.keytab
--principal=kudu/127.8.168.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.126:32929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.168.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:02:06.638288 30943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:02:06.639134 30943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:02:06.646750 30948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:06.646761 30951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:06.647269 30949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:06.647722 30943 server_base.cc:1034] running on GCE node
I20240501 20:02:06.648200 30943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:02:06.649116 30943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:02:06.650277 30943 hybrid_clock.cc:648] HybridClock initialized: now 1714593726650262 us; error 27 us; skew 500 ppm
May 01 20:02:06 dist-test-slave-tzct krb5kdc[30930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:02:06.657276 30943 init.cc:377] Logged in from keytab as kudu/127.8.168.126@KRBTEST.COM (short username kudu)
I20240501 20:02:06.658278 30943 webserver.cc:460] Webserver started at http://127.8.168.126:34221/ using document root <none> and password file <none>
I20240501 20:02:06.658877 30943 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:06.658947 30943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:02:06.659122 30943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:02:06.660959 30943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-0/data/instance:
uuid: "5ce15a22c69b446bae555e7ebd3af578"
format_stamp: "Formatted at 2024-05-01 20:02:06 on dist-test-slave-tzct"
server_key: "4366fa8a0ec412f85ff94e2072eafb98"
server_key_iv: "1fb8dd58c75f9613b84caa67f2f53186"
server_key_version: "encryptionkey@0"
I20240501 20:02:06.661415 30943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-0/wal/instance:
uuid: "5ce15a22c69b446bae555e7ebd3af578"
format_stamp: "Formatted at 2024-05-01 20:02:06 on dist-test-slave-tzct"
server_key: "4366fa8a0ec412f85ff94e2072eafb98"
server_key_iv: "1fb8dd58c75f9613b84caa67f2f53186"
server_key_version: "encryptionkey@0"
I20240501 20:02:06.664664 30943 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:02:06.667011 30958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:06.667968 30943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:02:06.668082 30943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-0/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-0/wal
uuid: "5ce15a22c69b446bae555e7ebd3af578"
format_stamp: "Formatted at 2024-05-01 20:02:06 on dist-test-slave-tzct"
server_key: "4366fa8a0ec412f85ff94e2072eafb98"
server_key_iv: "1fb8dd58c75f9613b84caa67f2f53186"
server_key_version: "encryptionkey@0"
I20240501 20:02:06.668169 30943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:06.689184 30943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:06.692175 30943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:02:06.692348 30943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:06.699151 30943 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.126:32929
I20240501 20:02:06.699388 31010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.126:32929 every 8 connection(s)
I20240501 20:02:06.700138 30943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-0/data/info.pb
I20240501 20:02:06.702739 31011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:06.706211 31011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:02:06.707737 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 30943
I20240501 20:02:06.707860 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-0/wal/instance
I20240501 20:02:06.708088 8865 external_mini_cluster.cc:1428] Setting key 694cd0a024ee38d275d3640a58c0d1b2
May 01 20:02:06 dist-test-slave-tzct krb5kdc[30930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.168.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.125@KRBTEST.COM" created.
I20240501 20:02:06.719234 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.708082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:32929 (local address 127.0.0.1:53430)
0501 20:02:06.708544 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.708592 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:06.709463 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:06.710160 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:06.710165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:06.710633 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:06.711046 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:06.711063 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:06.712079 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:06.712083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:06.712624 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:06.712635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:06.712895 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:06.713507 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:06.713540 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:06.715074 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:06.716951 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:06.716958 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:06.716964 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:06.717202 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:06.717454 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:06.717456 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:06.717457 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:06.717523 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:06.718128 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:06.718132 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:06.718253 (+ 121us) client_negotiation.cc:769] Sending connection context
0501 20:02:06.718399 (+ 146us) client_negotiation.cc:240] Negotiation successful
0501 20:02:06.718627 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":100,"threads_started":1}
I20240501 20:02:06.719246 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.709094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.8.168.126:32929)
0501 20:02:06.709374 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:06.709381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:06.709422 (+ 41us) server_negotiation.cc:407] Connection header received
0501 20:02:06.709775 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:06.709779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:06.709957 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:06.710114 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:06.711210 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:06.711948 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:06.712752 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:06.712919 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:06.715191 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:06.715208 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:06.715215 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:06.715238 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:06.716821 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:06.717307 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:06.717312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:06.717313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:06.717377 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:06.717599 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:06.717602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:06.717603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:06.718025 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:06.718171 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:06.718493 (+ 322us) server_negotiation.cc:299] Negotiation successful
0501 20:02:06.718774 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
I20240501 20:02:06.721414 31011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:02:06.722091 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.721856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36835 (local address 127.0.0.1:48576)
0501 20:02:06.721971 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.722032 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20240501 20:02:06.722244 30961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36835: connect: Connection refused (error 111)
I20240501 20:02:06.722980 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.722787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36835 (local address 127.0.0.1:48584)
0501 20:02:06.722886 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.722924 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20240501 20:02:06.723093 31011 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.125:36835: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36835: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.125.keytab.
Entry for principal kudu/127.8.168.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.125.keytab.
I20240501 20:02:06.731843 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.125:36835
--webserver_interface=127.8.168.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.125.keytab
--principal=kudu/127.8.168.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:32929,127.8.168.125:36835,127.8.168.124:43935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 20:02:06.772337 31011 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } attempt: 1
I20240501 20:02:06.773175 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.772895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36835 (local address 127.0.0.1:48596)
0501 20:02:06.773034 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.773093 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20240501 20:02:06.773842 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.773636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36835 (local address 127.0.0.1:48602)
0501 20:02:06.773740 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.773781 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20240501 20:02:06.773962 31011 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.125:36835: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36835: connect: Connection refused (error 111)
W20240501 20:02:06.837350 31018 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240501 20:02:06.837608 31018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:02:06.837639 31018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:02:06.837714 31018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:02:06.840754 31018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:02:06.840812 31018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:02:06.840844 31018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:02:06.840883 31018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:02:06.840915 31018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:02:06.842106 31011 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } attempt: 2
I20240501 20:02:06.842826 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.842554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36835 (local address 127.0.0.1:48614)
0501 20:02:06.842694 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.842751 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20240501 20:02:06.843576 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.843335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36835 (local address 127.0.0.1:48622)
0501 20:02:06.843452 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.843502 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20240501 20:02:06.843688 31011 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.125:36835: Network error: Client connection negotiation failed: client connection to 127.8.168.125:36835: connect: Connection refused (error 111)
I20240501 20:02:06.845012 31018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:32929,127.8.168.125:36835,127.8.168.124:43935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.125.keytab
--principal=kudu/127.8.168.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.125:36835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.168.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:02:06.845774 31018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:02:06.846521 31018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:02:06.853471 31023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:06.853753 31018 server_base.cc:1034] running on GCE node
W20240501 20:02:06.853956 31024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:06.854518 31026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:06.855010 31018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:02:06.855863 31018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:02:06.857003 31018 hybrid_clock.cc:648] HybridClock initialized: now 1714593726856984 us; error 34 us; skew 500 ppm
May 01 20:02:06 dist-test-slave-tzct krb5kdc[30930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:02:06.859508 31018 init.cc:377] Logged in from keytab as kudu/127.8.168.125@KRBTEST.COM (short username kudu)
I20240501 20:02:06.860357 31018 webserver.cc:460] Webserver started at http://127.8.168.125:44195/ using document root <none> and password file <none>
I20240501 20:02:06.860890 31018 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:06.860954 31018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:02:06.861124 31018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:02:06.862779 31018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-1/data/instance:
uuid: "c192f1ab3c5b4f4b8f68d07f75361c42"
format_stamp: "Formatted at 2024-05-01 20:02:06 on dist-test-slave-tzct"
server_key: "2e11a3b9eb81098a2df882898bf8fb76"
server_key_iv: "e87a3e6ebfaf2a531d132e7edf5bc3f3"
server_key_version: "encryptionkey@0"
I20240501 20:02:06.863230 31018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-1/wal/instance:
uuid: "c192f1ab3c5b4f4b8f68d07f75361c42"
format_stamp: "Formatted at 2024-05-01 20:02:06 on dist-test-slave-tzct"
server_key: "2e11a3b9eb81098a2df882898bf8fb76"
server_key_iv: "e87a3e6ebfaf2a531d132e7edf5bc3f3"
server_key_version: "encryptionkey@0"
I20240501 20:02:06.866348 31018 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240501 20:02:06.868366 31033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:06.869360 31018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:02:06.869475 31018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-1/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-1/wal
uuid: "c192f1ab3c5b4f4b8f68d07f75361c42"
format_stamp: "Formatted at 2024-05-01 20:02:06 on dist-test-slave-tzct"
server_key: "2e11a3b9eb81098a2df882898bf8fb76"
server_key_iv: "e87a3e6ebfaf2a531d132e7edf5bc3f3"
server_key_version: "encryptionkey@0"
I20240501 20:02:06.869556 31018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:06.887058 31018 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:06.889683 31018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:02:06.889848 31018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:06.896225 31018 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.125:36835
I20240501 20:02:06.896430 31085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.125:36835 every 8 connection(s)
I20240501 20:02:06.897186 31018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-1/data/info.pb
I20240501 20:02:06.899519 31086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:06.902727 31086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:02:06.906960 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 31018
I20240501 20:02:06.907095 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-1/wal/instance
I20240501 20:02:06.907338 8865 external_mini_cluster.cc:1428] Setting key 043b8993c1ab23a007d2a8a3a1d2d15c
May 01 20:02:06 dist-test-slave-tzct krb5kdc[30930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.125@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.168.124@KRBTEST.COM; defaulting to no policy
I20240501 20:02:06.914373 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.904410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.8.168.126:32929)
0501 20:02:06.904580 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:06.904584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:06.905247 (+ 663us) server_negotiation.cc:407] Connection header received
0501 20:02:06.906043 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:06.906046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:06.906105 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:06.906178 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:06.907672 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:06.908154 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:06.908971 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:06.909182 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:06.911270 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:06.911289 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:06.911292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:06.911317 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:06.912660 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:06.913115 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:06.913119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:06.913120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:06.913163 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:06.913406 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:06.913409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:06.913411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:06.913554 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:06.913636 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:06.914097 (+ 461us) server_negotiation.cc:299] Negotiation successful
0501 20:02:06.914214 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20240501 20:02:06.914943 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.904640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:32929 (local address 127.0.0.1:53438)
0501 20:02:06.905107 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.905143 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:06.905837 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:06.906337 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:06.906345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:06.906720 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:06.907496 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:06.907514 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:06.908278 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:06.908281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:06.908814 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:06.908825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:06.909014 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:06.909570 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:06.909596 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:06.911118 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:06.912782 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:06.912788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:06.912798 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:06.913023 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:06.913244 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:06.913246 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:06.913248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:06.913326 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:06.913649 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:06.913655 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:06.913879 (+ 224us) client_negotiation.cc:769] Sending connection context
0501 20:02:06.914049 (+ 170us) client_negotiation.cc:240] Negotiation successful
0501 20:02:06.914292 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":117,"threads_started":1}
I20240501 20:02:06.916179 31086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.168.124@KRBTEST.COM" created.
May 01 20:02:06 dist-test-slave-tzct krb5kdc[30930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.125@KRBTEST.COM for kudu/127.8.168.125@KRBTEST.COM
I20240501 20:02:06.923859 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.916645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36835 (local address 127.0.0.1:48626)
0501 20:02:06.916762 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.916776 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:06.916849 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:06.917437 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:06.917439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:06.917561 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:06.917712 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:06.917717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:06.918713 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:06.918716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:06.919058 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:06.919065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:06.919165 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:06.919685 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:06.919700 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:06.920876 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:06.922587 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:06.922591 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:06.922593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:06.922791 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:06.923055 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:06.923058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:06.923060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:06.923115 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:06.923455 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:06.923459 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:06.923549 (+ 90us) client_negotiation.cc:769] Sending connection context
0501 20:02:06.923638 (+ 89us) client_negotiation.cc:240] Negotiation successful
0501 20:02:06.923734 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20240501 20:02:06.924018 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.916738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48626 (local address 127.8.168.125:36835)
0501 20:02:06.917069 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:06.917074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:06.917095 (+ 21us) server_negotiation.cc:407] Connection header received
0501 20:02:06.917139 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:06.917144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:06.917304 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:06.917434 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:06.917810 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:06.918603 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:06.919156 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:06.919360 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:06.920990 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:06.921011 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:06.921020 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:06.921041 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:06.922477 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:06.922891 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:06.922894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:06.922896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:06.922972 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:06.923187 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:06.923189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:06.923190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:06.923361 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:06.923462 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:06.923702 (+ 240us) server_negotiation.cc:299] Negotiation successful
0501 20:02:06.923813 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":136,"threads_started":1}
I20240501 20:02:06.925506 31086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:02:06.926185 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.925918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:43935 (local address 127.0.0.1:54434)
0501 20:02:06.926065 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.926126 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20240501 20:02:06.926343 31035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
I20240501 20:02:06.927116 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.926945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:43935 (local address 127.0.0.1:54438)
0501 20:02:06.927028 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.927068 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20240501 20:02:06.927271 31086 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:43935: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.168.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.124.keytab.
Entry for principal kudu/127.8.168.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.124.keytab.
I20240501 20:02:06.931892 8865 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
/tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.124:43935
--webserver_interface=127.8.168.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.124.keytab
--principal=kudu/127.8.168.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.168.84:34183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.168.126:32929,127.8.168.125:36835,127.8.168.124:43935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240501 20:02:06.934846 31011 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } attempt: 3
May 01 20:02:06 dist-test-slave-tzct krb5kdc[30930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for kudu/127.8.168.125@KRBTEST.COM
I20240501 20:02:06.943724 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.935511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.8.168.125:36835)
0501 20:02:06.935620 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:06.935623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:06.935641 (+ 18us) server_negotiation.cc:407] Connection header received
0501 20:02:06.935798 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:06.935800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:06.935841 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:06.935907 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:06.936586 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:06.937077 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:06.937879 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:06.938021 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:06.940489 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:06.940506 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:06.940508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:06.940534 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:06.942321 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:06.942738 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:06.942740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:06.942741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:06.942777 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:06.942982 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:06.942984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:06.942985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:06.943131 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:06.943204 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:06.943477 (+ 273us) server_negotiation.cc:299] Negotiation successful
0501 20:02:06.943590 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20240501 20:02:06.943760 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.935401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36835 (local address 127.0.0.1:48628)
0501 20:02:06.935575 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.935593 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:06.935701 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:06.935952 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:06.935956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:06.936155 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:06.936464 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:06.936474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:06.937181 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:06.937185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:06.937772 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:06.937783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:06.937914 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:06.938980 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:06.938997 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:06.940356 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:06.942438 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:06.942442 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:06.942444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:06.942648 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:06.942845 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:06.942848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:06.942850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:06.942905 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:06.943216 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:06.943219 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:06.943325 (+ 106us) client_negotiation.cc:769] Sending connection context
0501 20:02:06.943448 (+ 123us) client_negotiation.cc:240] Negotiation successful
0501 20:02:06.943588 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20240501 20:02:06.944677 31011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:02:06.945443 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.945175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:43935 (local address 127.0.0.1:54442)
0501 20:02:06.945300 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.945355 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20240501 20:02:06.946202 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.945956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:43935 (local address 127.0.0.1:54456)
0501 20:02:06.946093 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.946140 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20240501 20:02:06.946348 31011 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:43935: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
I20240501 20:02:06.976481 31086 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } attempt: 1
I20240501 20:02:06.977329 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.977008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:43935 (local address 127.0.0.1:54472)
0501 20:02:06.977165 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.977233 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20240501 20:02:06.977484 31011 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } attempt: 1
I20240501 20:02:06.978201 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.977946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:43935 (local address 127.0.0.1:54484)
0501 20:02:06.978073 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.978134 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20240501 20:02:06.978353 31086 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:43935: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
I20240501 20:02:06.978492 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.977956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:43935 (local address 127.0.0.1:54476)
0501 20:02:06.978153 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.978206 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20240501 20:02:06.979172 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:06.978963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:43935 (local address 127.0.0.1:54492)
0501 20:02:06.979063 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:06.979112 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20240501 20:02:06.979305 31011 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:43935: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
W20240501 20:02:07.036581 31093 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240501 20:02:07.036855 31093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:02:07.036902 31093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:02:07.036983 31093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:02:07.040009 31093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:02:07.040062 31093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240501 20:02:07.040083 31093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:02:07.040100 31093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:02:07.040117 31093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:02:07.044095 31093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.168.84:34183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.168.126:32929,127.8.168.125:36835,127.8.168.124:43935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/krb5kdc/kudu_127.8.168.124.keytab
--principal=kudu/127.8.168.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.168.124:43935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.168.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 20:02:07.044941 31093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:02:07.045724 31093 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 20:02:07.046562 31086 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } attempt: 2
I20240501 20:02:07.047294 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.047027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:43935 (local address 127.0.0.1:54498)
0501 20:02:07.047166 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:07.047219 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20240501 20:02:07.047951 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.047731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:43935 (local address 127.0.0.1:54512)
0501 20:02:07.047832 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:07.047871 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20240501 20:02:07.048076 31086 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:43935: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
W20240501 20:02:07.052692 31101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:07.053208 31098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:02:07.053663 31099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:02:07.053992 31093 server_base.cc:1034] running on GCE node
I20240501 20:02:07.054445 31093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:02:07.054445 31011 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } attempt: 2
I20240501 20:02:07.055146 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.054899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:43935 (local address 127.0.0.1:54520)
0501 20:02:07.055030 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:07.055076 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20240501 20:02:07.055552 31093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:02:07.055814 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.055567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:43935 (local address 127.0.0.1:54522)
0501 20:02:07.055707 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:07.055753 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20240501 20:02:07.055936 31011 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.8.168.124:43935: Network error: Client connection negotiation failed: client connection to 127.8.168.124:43935: connect: Connection refused (error 111)
I20240501 20:02:07.056759 31093 hybrid_clock.cc:648] HybridClock initialized: now 1714593727056730 us; error 41 us; skew 500 ppm
May 01 20:02:07 dist-test-slave-tzct krb5kdc[30930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593727, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240501 20:02:07.059319 31093 init.cc:377] Logged in from keytab as kudu/127.8.168.124@KRBTEST.COM (short username kudu)
I20240501 20:02:07.060173 31093 webserver.cc:460] Webserver started at http://127.8.168.124:42419/ using document root <none> and password file <none>
I20240501 20:02:07.060671 31093 fs_manager.cc:359] Metadata directory not provided
I20240501 20:02:07.060755 31093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:02:07.060927 31093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:02:07.062561 31093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-2/data/instance:
uuid: "d4a1884690f74138a568e2b076eba1fc"
format_stamp: "Formatted at 2024-05-01 20:02:07 on dist-test-slave-tzct"
server_key: "28b52f16a8a478da2ccb908202ac5c21"
server_key_iv: "db2db1b60d7bb84c9b2a54e2c57ace36"
server_key_version: "encryptionkey@0"
I20240501 20:02:07.062984 31093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-2/wal/instance:
uuid: "d4a1884690f74138a568e2b076eba1fc"
format_stamp: "Formatted at 2024-05-01 20:02:07 on dist-test-slave-tzct"
server_key: "28b52f16a8a478da2ccb908202ac5c21"
server_key_iv: "db2db1b60d7bb84c9b2a54e2c57ace36"
server_key_version: "encryptionkey@0"
I20240501 20:02:07.066051 31093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240501 20:02:07.068097 31108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:02:07.069108 31093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:02:07.069213 31093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-2/data,/tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-2/wal
uuid: "d4a1884690f74138a568e2b076eba1fc"
format_stamp: "Formatted at 2024-05-01 20:02:07 on dist-test-slave-tzct"
server_key: "28b52f16a8a478da2ccb908202ac5c21"
server_key_iv: "db2db1b60d7bb84c9b2a54e2c57ace36"
server_key_version: "encryptionkey@0"
I20240501 20:02:07.069293 31093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:02:07.087033 31093 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:02:07.089520 31093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:02:07.089654 31093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:02:07.095794 31093 rpc_server.cc:307] RPC server started. Bound to: 127.8.168.124:43935
I20240501 20:02:07.096012 31160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.168.124:43935 every 8 connection(s)
I20240501 20:02:07.096777 31093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-2/data/info.pb
I20240501 20:02:07.099118 31161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:07.102260 31161 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:02:07.107031 8865 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu as pid 31093
I20240501 20:02:07.107126 8865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHyS6Hz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1714593353892612-8865-0/minicluster-data/master-2/wal/instance
I20240501 20:02:07.107368 8865 external_mini_cluster.cc:1428] Setting key 029f053c828e52f006e1baa82886760b
May 01 20:02:07 dist-test-slave-tzct krb5kdc[30930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593727, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.124@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
May 01 20:02:07 dist-test-slave-tzct krb5kdc[30930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.124@KRBTEST.COM
I20240501 20:02:07.113507 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.103865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53440 (local address 127.8.168.126:32929)
0501 20:02:07.103998 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.104001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.104625 (+ 624us) server_negotiation.cc:407] Connection header received
0501 20:02:07.105410 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.105413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.105463 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.105524 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.106666 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.107167 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:07.107885 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.108074 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:07.110240 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:07.110262 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:07.110265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:07.110293 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:07.111746 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.112254 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.112257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.112258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.112293 (+ 35us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.112564 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.112567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.112569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.112755 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:07.112842 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:07.113289 (+ 447us) server_negotiation.cc:299] Negotiation successful
0501 20:02:07.113386 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20240501 20:02:07.114208 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.104092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.126:32929 (local address 127.0.0.1:53440)
0501 20:02:07.104488 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:07.104522 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:07.105218 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:07.105662 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:07.105669 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:07.106033 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:07.106561 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:07.106573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.107285 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:07.107289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:07.107766 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:07.107775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.107979 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:07.108572 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:07.108597 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:07.110078 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:07.111849 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:07.111855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:07.111865 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:07.112166 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:07.112362 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:07.112365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:07.112367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:07.112486 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:07.112851 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:07.112855 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:07.113058 (+ 203us) client_negotiation.cc:769] Sending connection context
0501 20:02:07.113229 (+ 171us) client_negotiation.cc:240] Negotiation successful
0501 20:02:07.113486 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":94,"threads_started":1}
I20240501 20:02:07.115551 31161 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:02:07.116930 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.107990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54524 (local address 127.8.168.124:43935)
0501 20:02:07.108328 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.108334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.108409 (+ 75us) server_negotiation.cc:407] Connection header received
0501 20:02:07.108545 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.108549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.108678 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.108827 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.109666 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.110621 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:07.111239 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.111454 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:07.113648 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:07.113668 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:07.113677 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:07.113701 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:07.115369 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.115744 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.115749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.115750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.115799 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.116058 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.116061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.116063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.116252 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:07.116440 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:07.116616 (+ 176us) server_negotiation.cc:299] Negotiation successful
0501 20:02:07.116759 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":123,"threads_started":1}
May 01 20:02:07 dist-test-slave-tzct krb5kdc[30930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593727, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.124@KRBTEST.COM for kudu/127.8.168.125@KRBTEST.COM
I20240501 20:02:07.122324 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.116088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.8.168.125:36835)
0501 20:02:07.116225 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.116229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.116241 (+ 12us) server_negotiation.cc:407] Connection header received
0501 20:02:07.116272 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.116274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.116309 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.116372 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.116875 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.117366 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:07.117882 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.118067 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:07.119658 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:07.119675 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:07.119677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:07.119700 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:07.121003 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.121381 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.121385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.121387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.121440 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.121666 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.121668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.121669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.121796 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:07.121862 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:07.122115 (+ 253us) server_negotiation.cc:299] Negotiation successful
0501 20:02:07.122202 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20240501 20:02:07.122341 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.115966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.125:36835 (local address 127.0.0.1:48644)
0501 20:02:07.116069 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:07.116081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:07.116140 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:07.116395 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:07.116397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:07.116534 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:07.116680 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:07.116686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.117475 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:07.117478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:07.117785 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:07.117792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.117882 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:07.118266 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:07.118277 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:07.119539 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:07.121115 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:07.121118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:07.121120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:07.121281 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:07.121527 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:07.121531 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:07.121533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:07.121585 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:07.121873 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:07.121876 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:07.121980 (+ 104us) client_negotiation.cc:769] Sending connection context
0501 20:02:07.122086 (+ 106us) client_negotiation.cc:240] Negotiation successful
0501 20:02:07.122190 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
May 01 20:02:07 dist-test-slave-tzct krb5kdc[30930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.126@KRBTEST.COM
I20240501 20:02:07.123395 31161 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } has no permanent_uuid. Determining permanent_uuid...
I20240501 20:02:07.125947 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.118820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53446 (local address 127.8.168.126:32929)
0501 20:02:07.118924 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.118926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.118938 (+ 12us) server_negotiation.cc:407] Connection header received
0501 20:02:07.119030 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.119032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.119068 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.119127 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.119872 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.120475 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:07.121049 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.121229 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:07.123247 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:07.123265 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:07.123268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:07.123294 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:07.124616 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.124973 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.124976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.124977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.125018 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.125210 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.125213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.125215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.125357 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:07.125512 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:07.125685 (+ 173us) server_negotiation.cc:299] Negotiation successful
0501 20:02:07.125797 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
May 01 20:02:07 dist-test-slave-tzct krb5kdc[30930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593727, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.124@KRBTEST.COM for kudu/127.8.168.124@KRBTEST.COM
I20240501 20:02:07.129794 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.123789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:43935 (local address 127.0.0.1:54540)
0501 20:02:07.123950 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:07.123963 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:07.124043 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:07.124217 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:07.124219 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:07.124350 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:07.124478 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:07.124482 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.125116 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:07.125119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:07.125415 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:07.125421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.125510 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:07.125912 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:07.125923 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:07.127045 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:07.128692 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:07.128695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:07.128696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:07.128871 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:07.129080 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:07.129083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:07.129085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:07.129134 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:07.129379 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:07.129382 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:07.129479 (+ 97us) client_negotiation.cc:769] Sending connection context
0501 20:02:07.129577 (+ 98us) client_negotiation.cc:240] Negotiation successful
0501 20:02:07.129660 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20240501 20:02:07.130203 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.123865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54540 (local address 127.8.168.124:43935)
0501 20:02:07.123947 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.123950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.124019 (+ 69us) server_negotiation.cc:407] Connection header received
0501 20:02:07.124104 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.124107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.124140 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.124197 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.124572 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.125028 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:07.125503 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.125675 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:07.127163 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:07.127178 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:07.127181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:07.127204 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:07.128600 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.128967 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.128969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.128970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.129010 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.129202 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.129204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.129205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.129308 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:07.129367 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:07.129973 (+ 606us) server_negotiation.cc:299] Negotiation successful
0501 20:02:07.130067 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
May 01 20:02:07 dist-test-slave-tzct krb5kdc[30930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.168.125@KRBTEST.COM
I20240501 20:02:07.133175 31161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc: Bootstrap starting.
I20240501 20:02:07.134081 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.127118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.8.168.125:36835)
0501 20:02:07.127238 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.127242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.127297 (+ 55us) server_negotiation.cc:407] Connection header received
0501 20:02:07.127378 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.127381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.127419 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.127485 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.128192 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.128850 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:07.129439 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.129560 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:07.131530 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:07.131544 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:07.131546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:07.131565 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:07.132850 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.133185 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.133188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.133190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.133231 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.133418 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.133422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.133423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.133545 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:07.133705 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:07.133843 (+ 138us) server_negotiation.cc:299] Negotiation successful
0501 20:02:07.133941 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240501 20:02:07.135525 31161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:07.136256 31161 log.cc:824] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc: Log is configured to *not* fsync() on all Append() calls
I20240501 20:02:07.137182 8865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240501 20:02:07.138172 31161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc: No bootstrap required, opened a new log
I20240501 20:02:07.138123 30985 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5ce15a22c69b446bae555e7ebd3af578"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53446
I20240501 20:02:07.139245 31086 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } attempt: 3
I20240501 20:02:07.141005 31161 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } }
I20240501 20:02:07.141192 31161 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4a1884690f74138a568e2b076eba1fc, State: Initialized, Role: FOLLOWER
I20240501 20:02:07.141515 31161 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } }
I20240501 20:02:07.141917 31167 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } } }
I20240501 20:02:07.142043 31167 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:02:07.142640 31161 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc [sys.catalog]: configured and running, proceeding with master startup.
May 01 20:02:07 dist-test-slave-tzct krb5kdc[30930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.125@KRBTEST.COM for kudu/127.8.168.124@KRBTEST.COM
I20240501 20:02:07.147432 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.139623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:43935 (local address 127.0.0.1:54550)
0501 20:02:07.139725 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:07.139739 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:07.139812 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:07.140117 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:07.140120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:07.140294 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:07.140529 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:07.140536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.141421 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:07.141424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:07.141950 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:07.141960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.142061 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:07.142585 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:07.142600 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:07.144044 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:07.146008 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:07.146011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:07.146013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:07.146211 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:07.146451 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:07.146453 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:07.146455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:07.146504 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:07.146937 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:07.146948 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:07.147033 (+ 85us) client_negotiation.cc:769] Sending connection context
0501 20:02:07.147128 (+ 95us) client_negotiation.cc:240] Negotiation successful
0501 20:02:07.147255 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
W20240501 20:02:07.147946 31178 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240501 20:02:07.148108 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.139775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.8.168.124:43935)
0501 20:02:07.139910 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.139915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.139936 (+ 21us) server_negotiation.cc:407] Connection header received
0501 20:02:07.139970 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.139973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.140015 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.140098 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.140642 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.141313 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:07.142048 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.142218 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:07.144188 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:07.144206 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:07.144209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:07.144235 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:07.145875 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.146313 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.146317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.146319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.146364 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.146585 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.146588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.146589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.146763 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:07.146892 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:07.147806 (+ 914us) server_negotiation.cc:299] Negotiation successful
0501 20:02:07.147922 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240501 20:02:07.146111 31093 master_runner.cc:186] Error getting master registration for 127.8.168.126:32929: OK, instance_id { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" instance_seqno: 1714593726697056 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240501 20:02:07.148447 31093 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240501 20:02:07.148020 31178 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:02:07.150786 31086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42: Bootstrap starting.
I20240501 20:02:07.153223 31086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:07.154029 31086 log.cc:824] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42: Log is configured to *not* fsync() on all Append() calls
I20240501 20:02:07.155098 31011 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } attempt: 3
I20240501 20:02:07.156044 31086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42: No bootstrap required, opened a new log
I20240501 20:02:07.158886 31086 raft_consensus.cc:348] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } }
I20240501 20:02:07.159054 31086 raft_consensus.cc:729] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c192f1ab3c5b4f4b8f68d07f75361c42, State: Initialized, Role: FOLLOWER
I20240501 20:02:07.159345 31086 consensus_queue.cc:260] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } }
I20240501 20:02:07.159741 31181 sys_catalog.cc:455] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } } }
May 01 20:02:07 dist-test-slave-tzct krb5kdc[30930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1714593726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.168.126@KRBTEST.COM for kudu/127.8.168.124@KRBTEST.COM
I20240501 20:02:07.160027 31181 sys_catalog.cc:458] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:02:07.160434 31086 sys_catalog.cc:564] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:02:07.163750 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.155583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.168.124:43935 (local address 127.0.0.1:54566)
0501 20:02:07.155772 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0501 20:02:07.155787 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0501 20:02:07.155887 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:02:07.156253 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0501 20:02:07.156257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0501 20:02:07.156453 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0501 20:02:07.156690 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:07.156699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.157619 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0501 20:02:07.157623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0501 20:02:07.158183 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0501 20:02:07.158194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.158312 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:07.158817 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0501 20:02:07.158845 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0501 20:02:07.160310 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0501 20:02:07.162357 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:07.162361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:07.162363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:07.162589 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:07.162844 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0501 20:02:07.162847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0501 20:02:07.162849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0501 20:02:07.162903 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0501 20:02:07.163280 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0501 20:02:07.163283 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0501 20:02:07.163368 (+ 85us) client_negotiation.cc:769] Sending connection context
0501 20:02:07.163472 (+ 104us) client_negotiation.cc:240] Negotiation successful
0501 20:02:07.163596 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20240501 20:02:07.164271 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.155626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54566 (local address 127.8.168.124:43935)
0501 20:02:07.155757 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.155761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.155841 (+ 80us) server_negotiation.cc:407] Connection header received
0501 20:02:07.155989 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.155992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.156045 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.156131 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.156839 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.157514 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:07.158304 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.158817 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:07.160446 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:07.160464 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:07.160467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:07.160493 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:07.162231 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.162701 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.162705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.162707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.162756 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.162996 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.162999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.163001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.163165 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:07.163243 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:07.164022 (+ 779us) server_negotiation.cc:299] Negotiation successful
0501 20:02:07.164127 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20240501 20:02:07.163703 31018 master_runner.cc:186] Error getting master registration for 127.8.168.126:32929: OK, instance_id { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" instance_seqno: 1714593726697056 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240501 20:02:07.166050 31018 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:02:07.167187 31011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578: Bootstrap starting.
I20240501 20:02:07.169617 31011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578: Neither blocks nor log segments found. Creating new log.
I20240501 20:02:07.170426 31011 log.cc:824] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578: Log is configured to *not* fsync() on all Append() calls
W20240501 20:02:07.170624 31192 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:02:07.170713 31192 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:02:07.172197 31011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578: No bootstrap required, opened a new log
I20240501 20:02:07.174724 31011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } }
I20240501 20:02:07.174916 31011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce15a22c69b446bae555e7ebd3af578, State: Initialized, Role: FOLLOWER
I20240501 20:02:07.175232 31011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } }
I20240501 20:02:07.175750 31195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } } }
I20240501 20:02:07.176000 31195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:02:07.176342 31011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:02:07.176414 30985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:02:07.176514 30985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:07.177717 30985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } }
I20240501 20:02:07.178639 30985 leader_election.cc:288] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [CANDIDATE]: Term 1 election: Requested vote from peers c192f1ab3c5b4f4b8f68d07f75361c42 (127.8.168.125:36835), d4a1884690f74138a568e2b076eba1fc (127.8.168.124:43935)
I20240501 20:02:07.179559 31060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5ce15a22c69b446bae555e7ebd3af578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42"
I20240501 20:02:07.179575 31135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5ce15a22c69b446bae555e7ebd3af578" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d4a1884690f74138a568e2b076eba1fc"
I20240501 20:02:07.179756 31060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:07.179775 31135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:02:07.180974 31060 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce15a22c69b446bae555e7ebd3af578 in term 1.
I20240501 20:02:07.180974 31135 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce15a22c69b446bae555e7ebd3af578 in term 1.
I20240501 20:02:07.181337 30961 leader_election.cc:302] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce15a22c69b446bae555e7ebd3af578, c192f1ab3c5b4f4b8f68d07f75361c42; no voters:
W20240501 20:02:07.183533 31206 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 20:02:07.183604 31206 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 20:02:07.183838 31195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:02:07.184069 31195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [term 1 LEADER]: Becoming Leader. State: Replica: 5ce15a22c69b446bae555e7ebd3af578, State: Running, Role: LEADER
I20240501 20:02:07.184393 31195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } }
I20240501 20:02:07.186031 31195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ce15a22c69b446bae555e7ebd3af578. Latest consensus state: current_term: 1 leader_uuid: "5ce15a22c69b446bae555e7ebd3af578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } } }
I20240501 20:02:07.186131 31195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [sys.catalog]: This master's current role is: LEADER
I20240501 20:02:07.186319 30943 master_runner.cc:263] No leader master found from master 5ce15a22c69b446bae555e7ebd3af578
I20240501 20:02:07.186386 30943 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240501 20:02:07.193811 30964 tablet.cc:2369] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:02:07.195200 31214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:02:07.196206 31214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:02:07.212420 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.193200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.8.168.126:32929)
0501 20:02:07.193322 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.193326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.198461 (+ 5135us) server_negotiation.cc:407] Connection header received
0501 20:02:07.198657 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.198661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.198714 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.198765 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.201413 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.202141 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:07.204541 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.204780 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:07.206698 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:07.206720 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:07.206723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:07.206754 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:07.208306 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.209586 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.209590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.209592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.209649 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.210239 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.210242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.210244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.210415 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:07.210531 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:07.212124 (+ 1593us) server_negotiation.cc:299] Negotiation successful
0501 20:02:07.212245 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240501 20:02:07.214869 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.193904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.8.168.125:36835)
0501 20:02:07.194068 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.194072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.200041 (+ 5969us) server_negotiation.cc:407] Connection header received
0501 20:02:07.200209 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.200213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.200289 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.200344 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.203004 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.203709 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:07.205183 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.205386 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:07.209247 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:07.209270 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:07.209273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:07.209303 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:07.211153 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.211766 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.211770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.211771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.211825 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.212816 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.212820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.212822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.213000 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:07.213110 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:07.214585 (+ 1475us) server_negotiation.cc:299] Negotiation successful
0501 20:02:07.214708 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20240501 20:02:07.216601 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:07.194620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54572 (local address 127.8.168.124:43935)
0501 20:02:07.194728 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.194732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.208989 (+ 14257us) server_negotiation.cc:407] Connection header received
0501 20:02:07.209185 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.209189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.209247 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.209298 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.210188 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.210809 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:07.211579 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:07.211755 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:07.213274 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0501 20:02:07.213294 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0501 20:02:07.213298 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0501 20:02:07.213353 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0501 20:02:07.214960 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.215441 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.215445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.215447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.215499 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0501 20:02:07.215726 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0501 20:02:07.215729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0501 20:02:07.215731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0501 20:02:07.215884 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0501 20:02:07.215994 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:07.216326 (+ 332us) server_negotiation.cc:299] Negotiation successful
0501 20:02:07.216447 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240501 20:02:07.222643 31135 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc [term 1 FOLLOWER]: Refusing update from remote peer 5ce15a22c69b446bae555e7ebd3af578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:02:07.225289 31212 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:02:07.229576 31167 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ce15a22c69b446bae555e7ebd3af578. Latest consensus state: current_term: 1 leader_uuid: "5ce15a22c69b446bae555e7ebd3af578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } } }
I20240501 20:02:07.229700 31167 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:02:07.230022 31060 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42 [term 1 FOLLOWER]: Refusing update from remote peer 5ce15a22c69b446bae555e7ebd3af578: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:02:07.230892 31212 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:02:07.234180 31181 sys_catalog.cc:455] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ce15a22c69b446bae555e7ebd3af578. Latest consensus state: current_term: 1 leader_uuid: "5ce15a22c69b446bae555e7ebd3af578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } } }
I20240501 20:02:07.234285 31181 sys_catalog.cc:458] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:02:07.242756 31195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ce15a22c69b446bae555e7ebd3af578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } } }
I20240501 20:02:07.242892 31195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [sys.catalog]: This master's current role is: LEADER
I20240501 20:02:07.243223 31167 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ce15a22c69b446bae555e7ebd3af578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } } }
I20240501 20:02:07.243325 31167 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:02:07.243516 31218 mvcc.cc:205] Tried to move back new op lower bound from 7022975906695634944 to 7022975906550738944. Current Snapshot: MvccSnapshot[applied={T|T < 7022975906695634944}]
I20240501 20:02:07.246047 31219 mvcc.cc:205] Tried to move back new op lower bound from 7022975906695634944 to 7022975906550738944. Current Snapshot: MvccSnapshot[applied={T|T < 7022975906695634944}]
I20240501 20:02:07.246968 31212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ce15a22c69b446bae555e7ebd3af578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } } }
I20240501 20:02:07.247102 31212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578 [sys.catalog]: This master's current role is: LEADER
I20240501 20:02:07.248312 31181 sys_catalog.cc:455] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ce15a22c69b446bae555e7ebd3af578" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce15a22c69b446bae555e7ebd3af578" member_type: VOTER last_known_addr { host: "127.8.168.126" port: 32929 } } peers { permanent_uuid: "c192f1ab3c5b4f4b8f68d07f75361c42" member_type: VOTER last_known_addr { host: "127.8.168.125" port: 36835 } } peers { permanent_uuid: "d4a1884690f74138a568e2b076eba1fc" member_type: VOTER last_known_addr { host: "127.8.168.124" port: 43935 } } }
I20240501 20:02:07.248428 31181 sys_catalog.cc:458] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 20:02:07.248993 31221 mvcc.cc:205] Tried to move back new op lower bound from 7022975906695634944 to 7022975906550738944. Current Snapshot: MvccSnapshot[applied={T|T < 7022975906695634944}]
I20240501 20:02:07.257979 31214 catalog_manager.cc:1347] Generated new cluster ID: c9442c2daf024a6bbef72cab65676bd6
I20240501 20:02:07.258064 31214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:02:07.293691 31214 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:02:07.295095 31214 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:02:07.319955 31214 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5ce15a22c69b446bae555e7ebd3af578: Generated new TSK 0
I20240501 20:02:07.320899 31214 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240501 20:02:07.355010 31014 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:07.351878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53466 (local address 127.8.168.126:32929)
0501 20:02:07.352027 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.352032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.353793 (+ 1761us) server_negotiation.cc:407] Connection header received
0501 20:02:07.354036 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.354040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.354105 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.354166 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.354857 (+ 691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53466: BlockingRecv error: recv got EOF from 127.0.0.1:53466 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20240501 20:02:07.356554 31090 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:07.352617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48676 (local address 127.8.168.125:36835)
0501 20:02:07.352761 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.352766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.355623 (+ 2857us) server_negotiation.cc:407] Connection header received
0501 20:02:07.355805 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.355809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.355865 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.355920 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.356439 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48676: BlockingRecv error: recv got EOF from 127.0.0.1:48676 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20240501 20:02:07.361158 31165 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:07.353247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54588 (local address 127.8.168.124:43935)
0501 20:02:07.353361 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.353365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.356955 (+ 3590us) server_negotiation.cc:407] Connection header received
0501 20:02:07.360232 (+ 3277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.360237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.360293 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.360348 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.361018 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54588: BlockingRecv error: recv got EOF from 127.0.0.1:54588 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20240501 20:02:07.365693 31090 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:07.364069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48680 (local address 127.8.168.125:36835)
0501 20:02:07.364183 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.364187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.364698 (+ 511us) server_negotiation.cc:407] Connection header received
0501 20:02:07.364961 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.364965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.365044 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.365104 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.365588 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48680: BlockingRecv error: recv got EOF from 127.0.0.1:48680 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20240501 20:02:07.370580 31090 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:07.369210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.8.168.125:36835)
0501 20:02:07.369314 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.369318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.369731 (+ 413us) server_negotiation.cc:407] Connection header received
0501 20:02:07.369941 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.369945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.369995 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.370043 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.370480 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48684: BlockingRecv error: recv got EOF from 127.0.0.1:48684 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20240501 20:02:07.376646 31090 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:07.375247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.8.168.125:36835)
0501 20:02:07.375353 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.375357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.375801 (+ 444us) server_negotiation.cc:407] Connection header received
0501 20:02:07.375986 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.375989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.376039 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.376087 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.376542 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48686: BlockingRecv error: recv got EOF from 127.0.0.1:48686 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20240501 20:02:07.384425 31090 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:07.382815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48692 (local address 127.8.168.125:36835)
0501 20:02:07.382948 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.382952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.383471 (+ 519us) server_negotiation.cc:407] Connection header received
0501 20:02:07.383667 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.383671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.383726 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.383776 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.384304 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48692: BlockingRecv error: recv got EOF from 127.0.0.1:48692 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20240501 20:02:07.395879 31090 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:07.394785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48702 (local address 127.8.168.125:36835)
0501 20:02:07.394898 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.394902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.395112 (+ 210us) server_negotiation.cc:407] Connection header received
0501 20:02:07.395241 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.395243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.395291 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.395358 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.395778 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48702: BlockingRecv error: recv got EOF from 127.0.0.1:48702 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20240501 20:02:07.414850 31090 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:07.413800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.8.168.125:36835)
0501 20:02:07.413950 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.413955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.414051 (+ 96us) server_negotiation.cc:407] Connection header received
0501 20:02:07.414174 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.414178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.414235 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.414337 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.414726 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48708: BlockingRecv error: recv got EOF from 127.0.0.1:48708 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20240501 20:02:07.449739 31090 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:07.448656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.8.168.125:36835)
0501 20:02:07.448896 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.448900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.448921 (+ 21us) server_negotiation.cc:407] Connection header received
0501 20:02:07.449066 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.449069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.449128 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.449210 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.449595 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48720: BlockingRecv error: recv got EOF from 127.0.0.1:48720 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20240501 20:02:07.516641 31090 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:07.515660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48734 (local address 127.8.168.125:36835)
0501 20:02:07.515828 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.515832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.515928 (+ 96us) server_negotiation.cc:407] Connection header received
0501 20:02:07.516025 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.516028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.516081 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.516157 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.516517 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48734: BlockingRecv error: recv got EOF from 127.0.0.1:48734 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20240501 20:02:07.647687 31090 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:07.646625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.8.168.125:36835)
0501 20:02:07.646782 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.646785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.646902 (+ 117us) server_negotiation.cc:407] Connection header received
0501 20:02:07.647027 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.647029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.647080 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.647155 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.647543 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48742: BlockingRecv error: recv got EOF from 127.0.0.1:48742 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20240501 20:02:07.906769 31090 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:02:07.905682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48754 (local address 127.8.168.125:36835)
0501 20:02:07.905806 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:07.905811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:07.905973 (+ 162us) server_negotiation.cc:407] Connection header received
0501 20:02:07.906114 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:07.906116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:07.906165 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:07.906234 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0501 20:02:07.906660 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48754: BlockingRecv error: recv got EOF from 127.0.0.1:48754 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20240501 20:02:08.149823 31178 catalog_manager.cc:1259] Loaded cluster ID: c9442c2daf024a6bbef72cab65676bd6
I20240501 20:02:08.149920 31178 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc: loading cluster ID for follower catalog manager: success
I20240501 20:02:08.151540 31178 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc: acquiring CA information for follower catalog manager: success
I20240501 20:02:08.152669 31178 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d4a1884690f74138a568e2b076eba1fc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 20:02:08.172012 31192 catalog_manager.cc:1259] Loaded cluster ID: c9442c2daf024a6bbef72cab65676bd6
I20240501 20:02:08.172094 31192 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42: loading cluster ID for follower catalog manager: success
I20240501 20:02:08.173961 31192 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42: acquiring CA information for follower catalog manager: success
I20240501 20:02:08.175083 31192 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c192f1ab3c5b4f4b8f68d07f75361c42: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 20:02:08.423755 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:08.420793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.8.168.125:36835)
0501 20:02:08.421003 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:08.421006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:08.421021 (+ 15us) server_negotiation.cc:407] Connection header received
0501 20:02:08.421146 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:08.421149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:08.421265 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:08.421373 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:02:08.421705 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:08.422203 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:08.422786 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:08.422908 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:08.422995 (+ 87us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:02:08.423412 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:02:08.423511 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:08.423551 (+ 40us) server_negotiation.cc:299] Negotiation successful
0501 20:02:08.423598 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":64,"threads_started":1}
I20240501 20:02:08.429476 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:08.426463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54592 (local address 127.8.168.124:43935)
0501 20:02:08.426696 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:08.426700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:08.426712 (+ 12us) server_negotiation.cc:407] Connection header received
0501 20:02:08.426799 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:08.426802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:08.426922 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:08.427036 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:02:08.427339 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:08.427837 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:08.428445 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:08.428608 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:08.428691 (+ 83us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:02:08.429109 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:02:08.429215 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:08.429293 (+ 78us) server_negotiation.cc:299] Negotiation successful
0501 20:02:08.429341 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":66,"threads_started":1}
I20240501 20:02:08.435240 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:08.432154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48764 (local address 127.8.168.125:36835)
0501 20:02:08.432265 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:08.432268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:08.432503 (+ 235us) server_negotiation.cc:407] Connection header received
0501 20:02:08.432678 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:08.432681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:08.432765 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:08.432832 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:02:08.433172 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:08.433663 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:08.434528 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:08.434738 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:08.434786 (+ 48us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:02:08.434871 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:02:08.434946 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:08.435062 (+ 116us) server_negotiation.cc:299] Negotiation successful
0501 20:02:08.435113 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20240501 20:02:08.435271 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:08.432269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.8.168.124:43935)
0501 20:02:08.432371 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:08.432375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:08.432440 (+ 65us) server_negotiation.cc:407] Connection header received
0501 20:02:08.432588 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:08.432591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:08.432634 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:08.432705 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:02:08.433087 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:08.433772 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:08.434663 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:08.434798 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:08.434834 (+ 36us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:02:08.434896 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:02:08.434970 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:08.435098 (+ 128us) server_negotiation.cc:299] Negotiation successful
0501 20:02:08.435142 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20240501 20:02:08.435876 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0501 20:02:08.431970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53482 (local address 127.8.168.126:32929)
0501 20:02:08.432406 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0501 20:02:08.432410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0501 20:02:08.432425 (+ 15us) server_negotiation.cc:407] Connection header received
0501 20:02:08.432523 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 20:02:08.432527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 20:02:08.432661 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 20:02:08.432790 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0501 20:02:08.433234 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:08.433982 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0501 20:02:08.434733 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0501 20:02:08.434913 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 20:02:08.435030 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0501 20:02:08.435355 (+ 325us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0501 20:02:08.435445 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0501 20:02:08.435683 (+ 238us) server_negotiation.cc:299] Negotiation successful
0501 20:02:08.435738 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":63,"threads_started":1}
I20240501 20:02:08.439038 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 30943
I20240501 20:02:08.444389 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 31018
I20240501 20:02:08.449072 8865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHyS6Hz/build/debug/bin/kudu with pid 31093
2024-05-01T20:02:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2940 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5832 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (374565 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration
1 FAILED TEST